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1.0 


SUMMARY 


A number of studies have been conducted in an attempt to understand the complexity involved in 
andlrng an Air Traffic Control (ATC) situation. These studies have typically based their 
measurement of complexity primarily on the level of physical workload required by the Air Traffic 

n^ a rnn^?f 0rtUnat f y L many 0f . these r studies do not consider the cognitive requirements placed 
on the Controller, simply because this information is not easily measured. As the aviation 

nTn™ lty T VeS t ° Wards J a “ free m « ht ” environment, the complexities associated with ATC mav 
not necessarily increase or decrease, but they will most certainly change. Because this oroDosed ^ 
free fhgh ^environment will place the Controller in more of a monitoring role, the cognitive^ 
wmptexity associated with the Controller’s task will further change. Complexity^ it is perceived 
by the Controllers (who will still be ultimately responsible for traffic separation) will become 
mcreasmgly more important to understand. It is our position that an evaluation and understanding 
of the current and future ATC complexity would be best achieved through an analysis of the g 
costive tasks of the Controller (i.e strategies and decision making activities), and that of 
complexity may not be accurately reflected through measures of physical workload alone 

We begin this report with a description of some of the key results obtained from our examination 
and evaluation of ATC complexity. Following these defied findings, we will des^Se 

ana ,| ysesand analysis methods used to obtain these results. These analyses were based 
Sn,«H^ ne T rk f ° r r deVelop l ng “ d eval nating a model of the perceived complexity^ an air traffic 
PorfS W1 T SPeClflC regard t0 the characteristics that impact the cognitive abilities of the 

STtC id T c a n CfT' thls /amework does not depend on any spetific type of procedures 
tor ATC and can therefore be used to evaluate complexity in both cunrent and future ATC 

today’s System ^° wever ’ for the current stu dy, we do assume that airspace is sectorized! as it is in 

T S !y dy . include L the identification of the various characteristics, or factors of an air 
jraffic situation that impact the cognitive complexity of control, a complexity algorithm which 

™>cciT° r w S ^ e re , atlve and absolute weightings (assigned by Controllers during Focus Group 

and tH f evaluatl ° n of this complexity algorithm as presented in a number 
of ControUer-In-The-Loop simulations under both current and “free flight” procedures. 

Next, we present a description of how our initial complexity measurement was formulated As 

^ CTlbe the ltera i ive pr0CeSS We used to ^e measure to 
Confi-oUcrs perceptions of complexity. These complexity measure modifications were P primarilv 

r^.° n H feSUltS obta “? i durin g the simulation sessions and on the results of analyses of the 
complexity measure as applied to recorded live traffic situations. analyses or the 

With the completion of the development of the complexity measure, we are able to describe th* 
ai£ysi s of recorded traffic scenarios with the complexity measure. tt 

proced u S. , St^^p^S£ i being C0 " dUC,ed CUn ' e "'- 

Following the description of the complexity measure, the use of the measure is exnlorcd fhmncrh 

slJtem °Thff° Ut a P ^P° sed Dynamic Resectorization and Coordination Technology (DIRECT) 
oystem. The focus of this system would be to provide Air Traffic , 1 ' 

tod Jo evaluate future traffic patterns, based on the ^ 

wouM also suggest various complexity reduction stnde^rangS* tta^Sn^S / 

^ MlcSL and 
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will present a short summary of comments made by Controllers during these Controller-in-the loon 
simulations. The statements presented will no doubt be of particular interest to researchers and P 

ofth^StJre 1117611 ^ 7 W ° rking tOWardS transf ° rming current ATC s y stem t0 meet the demands 

Finally, we present a review of previous studies’ attempts at measuring complexity and a 

“* ,0 ^ PUCed of 
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2.0 


INTRODUCTION 


The motivation to develop and evaluate a model of air traffic complexity comes from the recent 
introduction of the free flight” concept and procedures for ATC (RTCA, 1995). Many 
descriptions of the free flight system state that safety will not be compromised. To ensure safetv in 
any complex system, it is necessary to understand the impact of any major changes in system 
procedures on the operators of that system. This understanding will enable us to develop 

W1 i TT 6 system beneflts ( in terms of safety and cost) without physically 
and/or mentally overloading the operators responsible for that system. y y 

flight ^ ® S1C demise of free flight is that Pilots can choose the most direct (and presumably optimal) 
flight paths to reach their destinations (RTCA, 1995). Under this definition, normal separation ' 
assurance and traffic routing will be the responsibility of the Pilots, while the Air Traffic 
specialists will assume a more passive, monitoring role. However, Air Traffic Specialists will still 
be expected to assume control under certain conditions.* The question is whetheror not the 
specialists will be able to easily and effectively intervene when needed. The answer depends in 
part, upon the complexity of the situation and the capabilities and limitations of the specialist.’ 

Other incarnations of free flight may not necessarily restrict the Controller to assume a primarily 
r ° ? 85 some interestm g work has detailed some difficult problems associated with 
rnitrrJifL system operator to quickly transition from monitoring a complex system to actively 
rrvm^h, (Endsley & Kiris, 1995). In any case, the fact remains that the overall 

complexity of the future ATC system will change. Indeed, it is likely that new complexities will 
also be realized. These complexities will exist both in the structure of the ATC environment and in 
the structure of the traffic itself. In order for us to move towards the best design for a future ATC 
system, it is important to be able to understand the cuiTent complexities, the ejected changes in 
cse romplenbes with the introduction of the future system, and the impact that those changes 

could have on the human operators of that system. ranges 

^e o^plexity of air traffic control is influenced by many factors, including the abilities of each 

WWte 1Cr ’ thC availabIe > md ^ complexities of the ATC environment itself 

While all of these aspects of complexity are important to understand, they are quite large in scope 

i*Hf de Th° f< ? US OUr Study T d . efme our measure of complexity based on the air traffic situation 
teelf. Therefore, in our evaluation of ATC complexity, we focus on the events or factors in a 

safe'll 1 ^ffic^3 < ^.c C0 " tr0 “ <:r ’ S PhySiCaI ‘“ d CQg " idVe pr ° CeSSK t0 * 

ZtaS £ m *“*"• rf * specmc "*** ■» chSrils 

M impacts* (problem solvmg, strategy formulation) and physical (communications 

P 1 1 8 ? on 1116 Controller. Controller input to the tfefinition of 

SS? a ! d r C % ext f nsive a” 100111 of knowledge of the domain. Consequently, this called for 

expat Air Traffic Specialists to be used in identifying and evaluating complexity factors and to 

£ simula *°y s desi $ ned t0 further develop the complexity measme. Although om 
aluation framework does not necessarily depend on any specific type of ATC procedures initial 

SSS.** mgh * ““ migh * oJZZSt used ,0 


As proposed. Controller intervention will only occur when: (1) tactical conflict resolutions are needed ('ll firm. 
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This rest of this report is organized as follows: First, we will present the detailed findings from 
our examination and evaluation of ATC complexity. Results of our study include the identification 
o the vanous characteristics, or factors, of an air traffic situation that impact the cognitive 
complexity of control and a complexity algorithm which incorporates the relative and absolute 
weighting 5 (assigned by Controllers during focus group sessions) of these factors. Additional 

ofFnntniw ° f ^ gonthm ^ based on “ evaluation of the algorithm as presented in a number 
mrS r fr ? U t 5 e " °°P simulatl0ns under both current and free flight procedures, and the 

made. 10 the complexity algonthm based on these findings. These analyses were 
based on an evaluation framework and consequently, the framework will also be described. 

With the completion of the development of the complexity measure, we are able to describe the 
analysis of recorded traffic scenarios with the complexity measure. This analysis evaluates the 
differences in complexity between air traffic being conducted under current, clearance-based 
procedures, and future free flight procedures. an Dased 

THie use of the validated complexity measure is explored through discussions about our DroDosed 
Dynamic Resectorization and Coordination Technology (DIRECT) System. The focus of this 
system is to provide Air Traffic Management personnel with a tool to evaluate future traffic 

on . the expected complexity of that traffic. The tool would also suggest various 
mplexity reduction strategies, ranging from the restructuring/redirecting of aircraft streams to the 

^ wi 11 atso present a review of other techniques that have been used for measuring 

SSofSSS focused on the measurement of physical actions as an 
.ca“ o n °f ATC complexity. However, because a Controller’s mental processes are also heavilv 
impacted by mcreased complexity, some illustrative examples are presented which support the Y 
argument that measures of physical processes alone are not enough in order to fully understand the 
comply of ATC. This section will also describe some of the difficulties ^scSwTth 
evaluating and measuring mental processes. This background data serves to provide a justification 
£ P t0 be phced °" underst “ d “S ^ cogmtive of cSol as ?n 

Finally, study conclusions and appropriate references will be included. As further support of the 
current stady effort, we will discuss some additional interesting findings 

on elites mt0 C ° mplexi,y of air »nd the impact that frefflight may 
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3.0 


DETAILED FINDINGS 


3.1 Individual Complexity Factors 

Thetable below shows the individual factors that are being computed in the complexity algorithm 
These factors were identified through a number of efforts including literature reviews ^Controller 

iUSwinTf ’ expl J )ra f° r y simulati ons, and Complexity Focus Group sessions. In addition to the 
dividual complexity measures, an overall complexity measure is also computed Complete 

MeSods° nS ** “* presented 111 Chapter 4 0 * Supporting Analyses / Analysis 


Aircraft Count 

[Acn 

Angle of Convergence in Conflict Situation 

[ANG] 

Number of Aircraft Climbing or Descending 

[CoD] 

Distribution of Closest Points of Approach 

[CPA] 

Aircraft Density 

[DNS] 

Level of Knowledge of Intent of Aircraft 

DNT| 

Neighbors 

[NBR] 

Proximity of Aircraft to Sector Boundary 

[PRX] 

Proximity of Potential Conflicts to Sector Boundary 

[PRx-q 

Airspace Structure 

[STR] 

Variance in Aircraft Speed 

[VAS] 

Variance in Directions of Flight 

[VDF] 


Table 1. Individual Factors Used in Complexity Algorithm 


3.2 The Complexity Algorithm 

CO S exit ^ dg0rithm ’ which has heen developed, verified, and validated using the 

Lissg?*** of 

The final complexity algorithm: 


0.0172 x ACT (MAX., 10.0) 
0.328 x DNS (MAX., 10.0) 
0.0498 x CPA (SUM, 15.0) 
0.1070 x ANG (SUM, 15.0) 
0.0426 x NBR (SUM, 15.0) 
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0.0754 x PRX-C (SUM, 15.0) 
0.1134 x CoD (SUM, 15.0) 
0.0709 x VDF (MAX., 10.0) 
0.0 x VAS (MAX., 10.0) 
0.2 x PRX (SUM, 10.0) 
0.0676 x STR (MAX., 10.0) 
+ 0.2564 x INT (MAX., 10.0) 

OVERALL COMPLEXITY 

Figure 1. The Complexity Algorithm 


ms ovendl complexity algorithm is a weighted sum of contributions from individual complexity 

t^o?i ff h^iSr Cnbed ab0VC ' jj* com Plexity factors contributes to the overall compfexity^ 

maximum (MAX ) function or a summation (SUM) function. A weighted sum 
(WEKjHT) function is also available, but was not used in the final complexity algorism Each 

SUM 1 ' ShJwiv 8 IT" 1 thC alg ° nthm above “ a Action of the contribution type, MAX or 
d * e !?? k ".^ ad tune* in minutes, over which the contribution function is applied The 

report UfeS ^ defme ^ dg0nthm wiU ** described in more detail in further sections of the 


3.3 Traffic 
Flight 


Complexity Analyses - Current Procedures Vs. Free 


The density and the number of closest points of approach associated with current procedures and 
free flight were compared on a sector-by-sector basis across 15 minute time intervals (for these 
STK S ° nS ’ TRAC0N was considered to be one sector). A count was made of all cases in 
CU ^ nt pro ? edures complexity was higher and cases in which the free flight comolexitv 

hH^ hlg TT, r ' 1116 r< f ults from a 6 hour System Analysis Recording (SAR) data sample are shown 
below. These results are accumulated over sectors and time intervals. Note that these results mav 

"hlnTwerei^orT ** ° ! ' '*** measurements because cases in which the complexity does not 



CP > FF 

FF > CP 

DNS 

293 

273 

CPA 

344 

338 


Table 2. DNS and CPA Comparison - Current vs. Free Flight 


^ 0 £ a ‘ a sknilar rea uHis obtained here as in a previous study conducted by the FAA and 

T Py n bum ' 1995) ' tetefo 

g p ocedures decrease sector density and conflict events as compared to current procedures. 
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A comparison of the overall complexity between current procedures and free flight was also 
conducted. These results, again accumulated over sector and time intervals, are S presented below. 



CP > FF 

FF > CP 

Overall 

145 

971 


Table 3. Overall Complexity Comparison - Current vs. Free Flight 


oJJ? at r f ^ ?° mpanSon mdicate a substantial increase in the number of cases in which the 
mJjHo ATC COmp ? X J y - 1S ? eater under free flight than under current procedures Note that man v 
fl V0 Ved m t^companson of overaU complexities than in the comparisons of Y 
density and conflict events. This is caused by the fact that many more traffic characteristics are 
cons^ered m the overall complexity measure, which results in fewer situations in which the 
complexity remains the same between current procedures and free flight. 

This result provides a very strong indication that measures of density and conflict events are not 
sufficiently representative of the overall complexity of ATC. The differences in traffic 
characteristics - other than density and conflict events - have a significant impact on the difference 
in overall complexity between current procedures and free flight cutterence 


3.4 


Complexity Reduction and the DIRECT System 


A number of complexity reduction heuristics have been identified in this study These heuristics 
can be grouped mto two major classes, based on the ATC element (air traffic or sector) affecte? 
Heuristics that affect the air traffic itself include: ° U attected. 

• Sector Avoidance 

• Changing Conflict Geometries 

Creating Aircraft Streams (speeds, headings, or climbing/descending aircraft) 

• Moving Conflicts. 

Heuristics that affect the structure of the airspace include: 

• Temporarily Moving Sector Boundaries 

• Increasing Airspace 

• Changing the Sector Shape. 

with an ^^^^l^tion'tov^^s^iffic'p^jlerns^lS^Dn^CT's^sb^ ^ r ^ lc - Spocialists 
provide Air Traffic Specialists " 

% appropriate 

oae of airspace, and to allow more aircraft to fly“Xfr£ ** 

atom the use of these heuristics, and the benefits of the DIRECT System are prodded fn siST 
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SUPPORTING ANALYSES / ANALYSIS METHODS 


4.1 The Evaluation Framework 

4.1.1 The Challenge of Applied Experimentation 

Today, many applied psychological experiments deal with the problem of trying to understand a 
large range of human-machine systems. These studies themselves can differ in complexity from 
operator manipulates menus of a window-based word processing system to how 
a Conh-oller effectively manages and controls air traffic. Whatever the focus of study, there are 
some fundamental problems with traditional methods of experimentation. As the human-machine 
system under study becomes more complex, these problems become more difficult to overcome. 

The main reason for performing an experiment is to be able to generalize what was learned in the 
experimental setting to some target setting. However, given the highly complex nature of some 
types of human-machine systems (such as Air Traffic Control), thifgoal of generalizable results is 
often difficult to obtain. What we have seen in the past, in terms of traditional psychological 

may T SUfFlCC “ a solution to this problem. For example, classfca! research 
methods (wherein the experimenter manipulates an independent variable and measures the resulting 

d0 not alwa y s “*» tato ««««• the many SSSSSSS? * 
relationships that exist between elements in a complex system. 

It is suggested that a greater understanding of the target work domain, along with a more careful 
e ection of subjects and tasks, is needed to better represent the operational setting of interest and 
at doing so will increase the validity of these generalized results. Our evaluation framework is 
Contr T ° n thCSe Suggestions and de P ends greatly upon participation from current Air Traffic 


Understanding systems of increasing complexity necessitates that new, more complex methods of 
analysis be used m order to handle the many possible interactions that can occT^oX 

fX? g C L° mp e f S ^ Ste ? S ’ 14 become increasingly difficult to interpret results. Our evaluation 

rk wasdesi S ned so that 11 could be used to create an experimental design that addresses 

achieve vZfgS^ble mtult US “ 8 ^ wc bdie » *>“' be better able to 

4.1.2 Work Domain Expertise 

One of the most important contributors to the usefulness of any research effort is a thorough 
understanding of the work domain under study. This understanding not only aids in identifying 
issues for study, but it also provides a baseline from which to begin analyzing experimental data 
and interpreting the results. Although this understanding of the work domain can be accomnlichpH 
£rm.gh many different methods, a trade-off does exist fie.. inc^ZZS £^ P “eb 

'Hierefore, the best and quickest way for us to uncover the detailed complexities associated with a ir 

r^-r 1 was to substantidl y Evolve Air Traffic Controllers in oufstudy. From a tmuT 
investment perspective, including classroom work, most en-route Con trolled reach Full 

^2^°? ^ VCl JP L) s !i tus r within 3 ' 4 y ears - Teiminal ControUem may t^e 5 to 6 years to 

reach FPL status. Tlie number of years required to be considered an FPL Controller reflect the 

iS f AT n ®f a basis for understanding the complexities of the work domain 

frotrfxwnenS com P>“ ti « of air ttaffic control must include input 

from experienced, FPL Controllers. Without this input, it is highly likely that the subdeties of the 
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red worSt Jstem 001 ^ inVeStigated ’ and ±e resultin g data will not be very representative of the 

Methods to obtain Controller input are quite varied in nature, and each method has certain benefits 
Mogford, Harwood, Murphy, and Roske-Hofstrand, 1994). Possible methods 
we considered for use m our study mcluded the collection of questionnaire data, unstructurS 
various protocol techniques, structured interviews, and simulations These 
“j- chosen the y are non-intrusive to actual ATC operations, are preferred by 

experts as being meaningful ways to elicit information, and in most cases allow direct access to 
Controller knowledge structures and cognitive processes (Mogford, et al„ 1994). 

4.1.3 Representativeness of Simulations 

The concept of designing psychological experiments to more closely represent the target 
operational setting is embodied by Bruns wik’s idea of representativeness (Bruns wik 1956) 

* V T eSS ? achieved r W,,cn y0U present Participant with an experimental’ situation that 

captures the relevant aspects of a corresponding real-life situation. The closer the design of the 
experiment is to the situations found in the actual operational setting, the more we can assume the 
e t0 f e generaUzed t0 that ^get concept. Brunswik (1956) also tells us that, in 
oI^^ SyC ^° gl ^ e ^ nnientation ’ “ dl relevfl nt external conditions are to be systematically 
controlled, and that all internal conditions are to be treated quasi-systematically by computational 
elimmation of random variability.” However, in the target operational setting^for example the 
of environment), this type of situation rarely exists because of die natural interaction 

elated and non-related variables and other factors influencing participants’ behavior and 
performance. Therefore, complex system studies should be desired so Aat ttey these 

interactions, and other influencing factors, in order to obtain a high level of represented veness. 

?n™!Xr i h { experime , ntal station that can present the subject with realistic representations of 

fractions is the use of simulators (Sheridan and Hennessy 1984) As 
abov f’ com P lex human-machine systems obviously cannot, by definition be addressed 
by ^ p e stnnulus-response experunents. Therefore, we must look towards more complex 
experimental designs, which may involve simulations. Various types of simulations can be built to 

situation ° f i^est, andsuch si^XZ^ provide us 
with methods to study complex behavior without being intrusive to the operators of the am ml 

199o! T* h0n - these simulations can increase the observabihtyof a system (Brehmer 

1990) and are potentially better for achieving a certain level of representativeness XXasS 

fhXifX ^, mp eXIty (“creased representativeness) corresponds to an increase in cost time and 
difficulty associated with interpreting the results. The Ultimate level of represeXtivXes’s of 

rEXn’^n meS f?? m atu dymg the actual operational setting. However, Sheridan and Hennessv 
(1984) tell us that using simulations (full-scope or otherwise) is one of the best wavs to studv^^ 

tUab ? S ’ ^ these simulations can best be used “to identify critical questions that later 

setti^s^’ m ^ mUCh m ° re eXpensive * time-consuming anTcomplexludX l o^onzl 
Our goal, then was to provide a simulation environment that represents the real-world ATP 

“«*« P ° SSib , le - 10 ° rder to d0 so > k was determined that pXi^one rft^most 

3?Sn^ P ^ tS t0 S T ula f was the communication between Controllers and* Pilots The 
?? te ™ USC ? U1 , dllS study Pseudo Aircraft System (PAS) developed by Syre 
subsidiary of Logicon) utilizes Pseudo-pilots at computer workstations In order to simulate 

eSSrUhJ 1 ! Pseudo-pilots received voice clearances and instructions from the Controllers and 

of A^r^to X?nwEd ^ SyStem Elated the dynamic response 

n *** cn kred clearance. In addition, the aircraft locations were oresenteH tn th* 

We*beli^ve ftat a t he < T^““ 0, l| diSPlay tha ‘ Very close,y ambles an actual Controller^dir display 
we believe that the increased representahveneaa of this aspect of the environment enabled ulto 


, a 
: the 
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more closely simulate a real-world ATC system than would 
not experience Pilot interaction. 


a situation in which the Controllers did 


4.1 .4 Complexity of Measurement 

° f ex P erimental desi gn translated into an increased complexity 
W1 < f l 116 m u yS1S ° f OUr results - Takin g into account the many different factors and 
disturbances that can be present in a complex system such as ATC, we were certain to see different 

suS, 1 ^ “ many differeM w “ ys - “ has by many that taSK,' Sdylle a 

subject s performance on a certain task, it is of primary importance to analyze the context ofthe 

1900 R n n m aCtl0n/decision P^ce (Sheridan and HeLessy, 1984- Brehlner 

R lk V 1 n 5 oiif° ray “L d Rote nberg, 1989; Sanderson, Verhage & Fuld, 1989) 

Further Brunswik (1956) states that when the complexity of the task and experimental setting 

Ci S th Udy 1S mcreased ’ the complexity of measurement methods must be increased accordingly 
^®" these suggestions, it was highly unlikely that traditional analysis methods, suchS statical 
measurements, would be suitable for understanding our collected data. Therefore we attemDted tn 

et aW 1994)2 where appropriate - However, we are reminded by Mogford, 

nr^-'icS mat statistical techniques cannot provide us with direct access to Controller cognitive 

HrSSfnH V ? was fundamental goal of this study. In addition, Mogford, Murphy R^ke- 
Hofstrand, Yastrop and Guttman (1994b) found that there was a high degree of correspondence 
between direct methods (such as questionnaires, interviews) and indirect methods (paired 
mpansons between factors) of identification of complexity factors. Therefore we concentrated 


4.2 Experimental Design 

0f i h f “P 6 ™ 141 design used for the study is organized as follows First a list 
of mrudamplex.^ factors, identified by the researchers at Wyndimere, wUl be presented ' This 
“r'f by a description of the initial, exploratory simulations 

A^°T P fr X1 c y U r GwU r P Sessions ” will be presented. This group consisted ^number ofm 

^n2^TCc2 S k^°L V h I T 1I f leV ff 1S °f xperie / lce md from Cerent control areas within 

our mea^^AcSc^riL^hn 116 ^ 6 “««"«* to the complexity factors used in 

asure. A description of how these weightings were assi gn e d will also he <*iven Finoiu 

we will describe the validation of our complelty measure too^lSations * 

4.2.1 Identifying Initial Complexity Factors 

T^e researchers at Wyndemere have extensive hands on experience working in operational ATP 
cilities. For example, m developing air traffic control automation tools Wvndemere ctaff h 
spent many hours working with Controllers, traffic management spe«iakteTd2S FA ? h 
^rsonnd m various ARTCC facilities and Terminal Radar Approach CONtrol (TR A rn?\n 

MihSb,^ Wyndemere staff members have also attended full Controller naming comses at 
ATCfacdmes m order togam or mamtrun an in depth undershmding of the operations at ^ 

mmmmm =ssr 
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complexity Input to these meetings also included reviews of existing studies and various Alt 
manuais Whteh provided a background of infomtation that could betsed as a bS idendWmt 
complexity factors These meetings resulted in the identification of a number of factors believed to 
mfluence the perceived evel of complexity of an air traffic situation. These initial complexly 
factors are presented below, in Table 4. P y 


Level of Knowledge of Intent of Aircraft (INT) 

Variance in Directions of Flight (VDF) 

Special Use Airspace (SUA) 

Performance Mix of Traffic (PRF) 

Weather Effects On Airspace Structure (WST) 

Number of Aircraft Climbing or Descending (CoD) 

Weather Effects On Aircraft Density (WDN) 

Distribution of Gosest Points of Approach (CPA) 

Aircraft Density (DNS) 

Angle of Convergence in Conflict Situation (ANG) 

Proximity of Potential Conflicts to Sector Boundary (FRX) 

Variance in Aircraft Speed (VAS) 

Number of Crossing Altitude Profiles (CAP) 



Table 4. Initial Complexity Factors, Identified by Wyndemere Researchers 


4.2.2 Exploratory Simulations 

W^mti de Sf y fk additi ° na ! complexit y factors * a number of simulations were held at 
Wyndemere For these simulations, current FPL Controllers participated in a real-time Controller 

'lt%^ TCS ,T l T nS u ° f b0th CUITent and free Procedures^ Fo/simulaticm pZoses 

dSiiSn ai^rt “ 8 “ Ch aircraft fly a direc ‘ route frora its departure import its ’ 

In the exploratory simulation sessions, the Controllers were presented with two scenarios that „,e 
cunren flight procedures and two scenarios that use free flight prot^uT^rdeS^bove ^e 

STS Denvei^ARTCC^Corri^u " C " r 5 “" ‘ ^gh alutude ^ui^souton 

cef nori? t £ wer ARTCC. Controllers were given flight strips for each of the flights in the 

scenario. These flight strips indicated an airway-based route of flight for the current procedure 
^? d a rif 601 r °!f e ° fflight for the free m « ht scenarios. The Controllers were^ked to 

lughly structured or as time critical m nature as the real-world ATC domain Vaarkoi * 

5^£sssrass^.s£T 
sasicasasassa: sssSSsr 

Ahhou^i the simulations were relatively informal, a number of interesting initial results h^ n 
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The first plot below (Figure 2) shows the latitudinal / longitudinal flight paths that would be 
followed by aircraft in one of the free flight scenarios, if no maneuvers were instructed or executor! 

J hls u scenano was designed to be the most complex free fhghfscenario whh 
almost all of the aircraft in the scenario approaching a very small area of airspace at the same tilf 

^« ure ^ leaves out two critical dimensions of the four dimensional scenario-altitude 
and time. This particular scenario presented all aircraft at flight level 350 so it is nnt „- 0 de t 
show a graph of aircraft altitudes. Still, the conflict s ttuatioT oulnot S pr^lv XtST , 

^rE, nta T “(o' 1 me d, "' ension - Fl « ure s 3 and 4 show the longitudinal aL UdutdtaaT * 
coordmates of each flight as a function of time, respectively. 



FigUre 2 1X11 ' ^ Flight Tracks of Aircraft Under Simulated “ Free Flight" Procedures 
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Figure 3. Longitudinal Position vs. Time for Simulated “ Free Flight ” Aircraft 



Figure 4. Latitudinal Position vs. Time for Simulated “Free Flight" Aircraft 
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?* ure 1 S ^° WS ? e scenano as controUed by one of the simulation subjects Again the clot 

^^^ongitudmal positions of each aircraft. Of interesting note in this scenario is 

^ at i^ e C °Tk used . on l y 15 vectors t0 resoIve ^ conflicts within a 10 minute time 
^ sc V ssed above ’ 11 18 dear that the physical task time required for the Controller to 

SES2* *** P lan not cxcessive-a fact that supports our claim that measures of physical task 
time alone are insufficient indications of complexity. ” ^ 



Figure 5. Lat / Long Flight Tracks of Aircraft With ATC Commands Issued 


filiwvf 6 f? 1 Sh c W the tcHgitudinal and latitudinal coordinates of each flight, as controlled as a 
CHaVA ° tUTie ‘ a P roce< ^ ura t standpoint, it is important to notice that UAL670 and 
COA321 were within 5 miles of each other at approximately 314 seconds intoTsimuktion. 
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Figure 6. Longitudinal Position vs. Time for Controlled Aircraft 



Figure 7. Latitudinal Position vs. Time for Controlled Aircraft 


■un acceptable 
itself. Since most 
, they were not 


that ControUer subject allowed these two aircraft to violate the minii 
action requirement may have been the product of the simulation envtaZ 
Conh-ollers are not required to provide protocol data during real-tXS^atio 
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entirely practiced at doing so for our simulation purposes. However, as mentioned above these 
were merely exploratory simulations and the key to the success of these simulations was to 

confrolhng fraSc^ “ P ° SSlble about what the Controller was doing and thinking while 

mterating result of the exploratory simulations was the significance that Controllers 

h W ^ 8 l ° f ?!“£ ° f lack thereof ’ of **craft. In simulation debriefings with 
feS® 1 * 11 sr subjects th e y felt that a strong increase in the level of complexity of the traffic 
situation would result if the aircraft were to actually maneuver on their own. As stated by the 

SiK T wm have 5^% of VFR flight while being offered 
irK protection (RTCA, 1995). However, since the Pseudo-pilot was controlling as manv as 

twelve aircraft, it was not possible for the Pseudo-pilot to determine what maneuvers would be 

realistic for each individual aircraft to make, in order to exploit that flexibility In essence this 

created a situation in which the scenario was no longer a true free flight scenario, as the Comrollers 

were able to trust that the aircraft would not alter their flight paths without Controller clearance. 

s till the presence of an operational error (with respect to separation minima) is an interesting issue 

T i c ? rar mi « ht "W* ^at the Controller 

limits of his abilities. Although this might have been confounded with the verbal protocol 
procedure the high level of complexity of the scenario may have left very little mental resources 
available to provide this verbal protocol data along with acceptable control performance. 

4.2.3 Critical Decision Interviews 

? aJ “ Pt t0 V u rify T c ? m P lexit y factors for inclusion in a complexity measure as well as to 

mSlght 1° 0ther fact ° rs which might contribute to traffic complexity (see Table 5) a 
Wyndemere researcher organized a meetmg with a TMC from the Denver ARTCC facilitv During 

™ C P arti cipat ed in an interview session that was based onthe CiS LS 
Method for knowledge eheitation, developed by Klein, Calderwood, and MacGregor noso? Tn 

fret^r ^ ^ WaS t0 idenbf y P ast scenarios that due to the 

fact that they were high in complexity. As part of the interview, the TMC was ask^tod^criS 

is go and expectancies for each situation, the cues he used for action, the actions he took the 
oto available options and the explicit factors responsible for making the sreSio The 

m,ps of Denver ^ f ■» ZSSfoSL 

R(»ilte from the critical decision interview meeting reinforced the importance of a Controller 

thC mtC ri of , other air craft. In addition, the TMC described the complexities 
associated with certain air traffic situations and the impact they have on the amount of avn?lnhip 
mrspace for in* by a ControUer. For example, from the maZw toSSthat 

JTM 0n , a secto f “ fact thaI 8 certain area of the sector is no longer available for 

use. In effect, the volume of usable airspace within a sector decreases in size when a weather cell 

h- ? 1S H m tur t affeCtS 4,16 overa11 density of aircraft distributed throughout the rest of the 
sector, which reduces the amount of freedom the Controller has for aircraft routing 

to t *5 Se fa P t0 /' s ’ the ™ c described the impact that multiple conflicts occurring in a 
short time period might have on the perceived complexity of an air traffic situation Tn hif 

r lve 

gtSsTw^t,^uoVe“lto a ) C ‘ ** ^ inV ° 1VKl m conflicting 
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Weather impacts the amount of usable airspace, and therefore the structure (size and 
shape) of the sector. This measure will examine tire impact that a weather cell can 
have on the structure of a sector, and how that transit into i 


| Proximity of Potential 
Conflicts to Sector Boundary 
fPRXl 

Aircraft Density 
[DNS1 

Number of Facilities 

[FAC1 

Number of Aircraft 
Climbing or Descending 

[CoDl 

Number of Crossing 
Altitude Profiles 

[£*£] 

Weather Effects On Aircraft 
Density 
[WDN] 


An examination of the location(s) of the potential conflict(s) with respect to current 
sector boundaries. 

A measurement of the density of aircraft with respect to the usable amount of 
airspace. 

A count of the number of facilities served by. or contained within, a specific sector. 


A simple numerical count of the number of aircraft expected to climb or descend 
altitude. 


m 


This measure is an examination of the number of ascending and descending aircraft 
profile pairs that are expected to occupy (in crossing) the same altitude within a 
specified period of time in the future. 

Weather also impacts the density of the aircraft in the sector, because the amount of 
available airspace is effectively reduced. Therefore, this measure will examine the 
impact that a weather cell has on i ' 


Variance in Aircraft Speed 

~ VAS 

Variance in Directions of 
Flight 
VDF 

Penance Mix of Traffic J A measurement that looks at the variance in performance capabilities of current and 
l rKr ] I expected aircraft 


A measurement that looks at the variability of speed tracked far each aircraft. 

A measurement that looks at the variability of direction for each aircraft to be 
controlled. 


I Winds 

[WND 

Distribution of Closest 
Points of Approach 

[CPA 

Angle of Convergence in 
Conflict Situation 
[ANG 
| Neighbors 

[NBR 

I Level of Knowledge of 
| Intent of Aircraft 
[INT 

Separation Requirements 
[SEP- 
[ Coordination 
[CRD 


A measure of wind speed and azimuth by altitude, and its impact on aircraft 
performance characteristics. 

This measure is a time-based distribution of the number of intersecting (laterally) 
flight paths which could be potential conflicts. 

A measure that examines the predicted angle of convergence in a conflict Shallower 
angles of convergence result in a longer period of potential conflict. 

Theproximity in lat. and vert, distance between ACFT pairs in conflict and other 
AIM wi thin some parameter distance or tim* 

A measure that looks at the effects that the knowledge of intent of an aircraft has on 
the complexity of a conflict involving that aircraft. 

A measure that examines the impact that imposed separation requirem^TfoT 
longitudinal sequencing and spacing has on complexity 

The impact that the presence of aircraft that require some form of coordination (jrith" 
L atere, etc ) for proper control has on the complexity of an air traffic sihiarinn 


Table 5. Complexity Factors Examined For inclusion in Complexity Algorithm 
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After the Critical Decision Interviews, the complete list of complexity factors was compiled and 
redundant factors were removed. In total, there were now 19 factors believed to contribute to the 

sltuation - T* 1 ^ factors, including the additional factors 
i d u U ^ i 60 * ^ ™ C mterview and die simulations, are those listed above in Table 5 

the Group ° f ^ fac,or 8 iven *° each ** Traffl <= SpeciaUst in 

4.2.4 Complexity Focus Group 

The next step in our process was to assign weighting values to each of the complexity factors In 
order to properly assign these weightings, we held a number of sessions in which we presented 
our complexity factors to 10 current, FPL Air Traffic Specialists (5 TMCs/Supervisors, 5 

ZTZ ’ aU ? enver The Specialists were asked to rate and rank the complexity 

factors in a number of ways. Since the overall impact of each of the individual factors may depend 
on the addition of other factors, die multiplicative effects between factors and even within muftiple 
occurrences of the same factor, if appropriate, were also examined. P 

diesesessions.each Air Traffic Specialists first participated in a Factor Interview which 
k designed to elicit both qualitative and quantitative information, as well as to aid in establishing 
a common language between researchers and Air Traffic SpeciaUsts with respect to the g 

definitions and assumptions associated with each complexity factor. Participant^ were then asked 
* ra * both * e m dmdual factors and factor pairs in terms of their absolute level of contribution to 
the perceived comptoxity of an air traffic situation. In addition, the participants were S to ra^k 
Ae factors against themselves m order to understand the relative relationships between these 

fhp SnH md h0W ^aT re i atl0nship affect the P erceiv ed complexity. The results from this part of 
the study were used to aid in assigmng the weighting values of each factor for use in the 
complexity measurement, 

4,2.4. 1 Factor Interviews 

55?“^ G f roup sess f on began with an interview; the purpose of which was to achieve a number 
SnS cf FU ^l?k m ^ n r 10ned ab °y e ’ interview was designed to familiarize the Air Traffic 
^ hith Wlth lefilu J tl< ? n “ d assumptions associated with each complexity factor In doing 
so both the researcher and the Air Traffic Specialists were better able to communicate their ideas 

anmred.^ 6 Speciflcs of each factor * and questions (posed by both parties) were more readily 

Second the interviews were designed to elicit both qualitative and quantitative information about 
SSi J 6 factors, and the forniat resembled the Critical DecSom E 

^i?H^ ^ d i abOVe ' 7116 coUected qualitative information was very valuable in that it 
helped determine useful starting points to begin analyzing the coUected quantitative date ^ well 
given the complex nature of the ATC system and the high degree of SS in h^Int^oHnn 
with that system, the qualitative data was essential to truly understand the details regarding the 
2? 1 ? system Examples of the type of data coUeS to Sr Svtws 

about the range of parameter values (of each factor) that impact complexity a 
to ^ unpact of this factor on the complexity of control and P ^ 
dependencies of a specific factor on other elements of air traffic control. 

Complete summaries of the results from the Factor Interviews are presented in Appendix B 
cnZIZ' “ ex r pl l ofthis data is presented l* 1 ™- This example is a SSuSyrffi 
[CoD] h£ r o®^mpl™rof^?o| nUraber ° fa,rCrafteXpeC,ed “ ^mg or descending 
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In general, Controllers agreed that an increase in the number of aircraft climbine or 
descending within a sector results in an increase in the complexity associated with 
the control of that sector. However , the relationship between the number of aircraft 
climbing or descending and complexity depends on the density of aircraft the J 
number of conflicting altitude pairs, the intentions of the aircraft, and the type of 
sector being worked. The answers given in the interviews were given with the 

rnnTr tl ° n the - Co " troller was irking an overflight sector, and some existing 
condition was forcing the aircraft to have to climb or descend. In general 8 

owever, most Controllers stated that if they were controlling an arrival or 

sector ’ then the impact of an increase in the number of aircraft climbine or 
descending wouldn t be as great as if they were working an overflight sector. 

to &l e a ran 8 e °f the number of aircraft climbing or descending that 
hey consider to be very high, high, and low in complexity, most answers were 
given in terms of the percentage of the total number of aircraft. Obviously then 
these numbers depend on the total number of aircraft within the sector (the density 
of the aircraft). Therefore, Controllers assumed “moderate" levels of traffic when 
stating their answers, presented below. J JJ 



n 

a M> 

Very Hiqh 

>52% 

14.6 

j Hiqh 

>31% 

ko 

00 

Low 

<23% 

6.5 


In addition, one Controller mentioned that as the percentage gets closer to 100 it 

actually becomes slightly easier again because in that situatiol, evenai, craft i is 
behaving in the same general manner. “"cruji is 


With respect to time, Controllers feel that looking ahead about 15 minutes is 

impac ‘ ** dimbing ° r demnding aircn * wm *»• on 

through rating and ranking scales, completed by each Air Traffic Specialises described Slow. 


4.2. 4. 2 Rating and Ranking Questionnaires 
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1 Absolute | 

Factor 

p 

o'"’ 

INT 

7.9 

2.18 

DNS 

7.2 

2.39 

CAP 

7.2 

2.04 

NBR 

6.7 

2 . 11 

CRD 

6.7 

2.45 

CPA 

6.5 

1.78 

CoD 

6.4 

2.07 

SEP 

6.3 

1.70 

PRX 

6.0 

1.94 

ANG 

6.0 

fr“ * 
00 

STR 

5.2 

2.66 

VDF 

5.1 

2.13 

PRF 

5.1 

2.51 

FAC 

5.0 

2.49 

VAS 

4.3 

2.31 

SUA 

3.9 

2.02 

WND 

3.2 

1.75 


Table 6. Absolute Complexity Ratings, Sorted In Descending Order 


Sr, S S°fo d , qUeSti ?. nnaire each ^ Traffic Specialist to rate the different combinations of 
pairs of factors in the same manner as above. In this case, they were to assien a rati n a nf « i rr * 
pair of factors that thev feel, when rnmhin^ assign a rating of 10 to a 


i* <0 a‘ S ' 


assign a rating of ' 

, „ _ .. * — umi uicjr icci nave vsncutue impact on the comnlevi tv 

a situation 8 ° l ° & paif ° factors that ^ feel onl y SQmewhat impact the overall complexity of 

^M^A dU m t0 #? e J^ 6 T°^ nt of data 00116016(1 from this rating questionnaire (171 ratings for each 

*1 ass?# th h S r mari2e ? ** tebles « P^ted - Appendix B^However^ 

^ofiles Xor rr API ^ K he abS0l u Ute oom P lexit y rating data from the number of crossing altitude 
p ofiies factor [CAP] combined with every other factor is presented below, in Table 7. 8 

th * Specialists were asked to rate the relative contribution of 

acn or the listed factors, against all others. For example the factor that th^v 1 v, QC tu* 

th^rating* of “ l^Thaf ^ ^ tr> ?' C situation ( above ’ all other listed factors) was to be'Sven' 

tomtunhfgfacL) wL to'S WSSSSES gT-4 "S', 

‘szi??, tv?*- pr r en,ed beiow sorted ** z to St 11 

^Invet^r^hT^Eown 
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Absolute 


Factor Pairs 

n 

o'-" 

CAP x INT 

7.8 

1.48 

CAP x DNS 

7.6 

1.96 

CAP x PRX 

7.3 

2.21 

CAP x CoD 

7.1 

2.13 

CAP x PRF 

7 

1.94 

CAP x CPA 

7 

2.91 

CAP x NBR 

6.7 

2.87 

CAP x SEP 

6.7 

1 1.77 

CAP x ANG 

6.6 

2.59 

CAP x VAS 

6.4 

2.91 

CAP x CRD 

6.1 

3.07 

CAP x VDF 

5.7 

3.43 

CAP x STR 

5.6 

2.37 

CAP x FAC 

5.3 

3.16 

CAP x WND 

5.2 

2.90 

CAP x SUA 

4.8 

2.66 


Table 7 Comb ^ed Rating Data For [CAP] With Other Complexity Factors 


f Relative I 

Factor 

z Score 

WDN 

1.14 

WST 

1.01 

INT 

0.64 

DNS 

0.57 

Cod 

0.46 

CPA 

0.41 

CAP 

0.23 

PRX 

0.16 

ANG 

0.12 

CRD 

- 0.14 

NBR 

- 0.18 

SEP 

- 0.21 

VDF 

00 

CM 

O 

1 

STR 

- 0.37 

PRF 

- 0.41 

SUA 

- 0.60 

FAC 

- 0.68 

VAS 

- 0.75 

WND 

“ 1.12 


Table 8. Relative Ratings Between All Individual Factors. Sorted by Z Score 
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Both the qualitative data and the rating and ranking data were collected in order to gain some 
msight as to the appropriate weightings that should be assigned to each factor in the complexity 
measure As well, the ranking data allowed us to determine which factors were considered to be 
the most important with respect to air traffic control. It was not realistic to assume that thTimoZt 
tha two factors would have on complexity would simply be an additive effect, based on £e P 
individual factor ratings. Therefore, the combinations of factors enabW us“ ex^ if ^td how 
the weightings might change when two factors are combined. 

The Factor Interviews provided additional detailed information about each factor and the data fm m 
these interviews was used to support the numerical data with respect to how and ’when the 

oartSinnni be As well > the interview data was useful in understanding how 

participants might answer a question (i.e., how does factor “x” impact complexity) differentlv 

based on the context in which that question was posed. The results from the data analysis and how 
prSentSnext 01 * 1065 ° f mformation were used k imputing the actual complexity measure are 

4.2.5 Controller -In-The-Loop Simulations 

The major portion of our complexity measure development took place during Controller-in-the- 
loop simulations. Results from these simulations were used to modify the complexity algorithm 
com^? 17 ' F ° r thC sm julaUon sessions, there were three differentcondition S P unde? which^r’ 

Flight, (C)U ' ren *’ 

4.2.5. 1 Conditions 

fSS^ 

tofollow^^^ e ^ute)^^^^a^^he^,tie^^through t ^^^^ation^(»l W In C additio^to lm 

taXS °" d f f s, * nated and providing dte controller with fuTfligta Sfp 

T cra c wer u requlred 10 finest ATC clearances for any desired routine chaLes - as 
y s system. For the scenarios presented under these conditions actual SAR flight t™ rk Hat a 

:l:n n rs. p z3s use, ,he: sari 

Admittedly even though the ament air traffic situation in any given sector will have been 
influenced by previous control decisions, the instantaneous complexity experienced within a 

A) n <?!^ 

7T. a" *11“" i “e wm be 

Controller nt Sector A will not be separating his/her aircraft teed on the tiafficSs ta Sector 
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Figure 9. Responsibility For Conflict Resolution Within a Given Sector 

( mil Eree Bight Procedures . During the complexity focus group interviews, the discussions 
about aircraft intent information were very interesting. Every Controller felt that if s/he did nnt 
have aircraft intent information (with respect to changes in speeds, altitudes headings etc ) then 
W ?, Ul T d beCOm ? vef y hi « h - I" one Controller’s opMon,Sol^ 
mfiruteiy harder. In general, it is believed that Controllers may have a difficult time imaeininv a 

<ie - 10 • 15 " ^ ivsx 

Jh»n^ te ^ • incr ! ased flexibility, an additional Pseudo-pilot was used to input these 

helieve that scn P te ^’ ^craft- by-aircraft basis. In allowing this increased aircraft flexibility we 
beheve that this significantly impacted the complexity of control based on the lack of short tL 

5225~ “ rs placed on * e 

T t 1 ' Ha “ Fr “ Fli * hl condition was designed so that Controller still had , 

additional Pseudo-pilot was again used to inout “nilot d^cirwi” aircraft ^ Tnerefore, an 

(SU^^ 
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4.2.5. 2 Scenarios 

A total of thirteen (13) test scenarios were generated for use in the simulations Four (4) scenarios 
were presented under the (C)urrent Procedures condition, six (6) under the (H)alf Free Flight 

**** & under die (F)ull Free Flight condition. In addition, each scenario had one 
of three different levels of complexity, as computed by our algorithm. Each Controller was 

fn c 11 !^ WU i, tW L ° ® Cu f rent Procedure scenarios, two (2) Half Free Flight scenarios and one 

( ) ull Free Flight scenario. The order of the scenario presentation, as well as the level of 

scenar 10 complexity , w as ; randomized across Controllers Scenarios were firsTgeneSSd by 

Wvndemere^^CATf^ * f Com P lexit y Analysis Tool (CAT), developed by 

nf i‘ too1 enables us to view vanous traffic scenarios, along witha computation 

of the complexity of that scenano, to identify specific situations which can be used for simulation 

r3nl? S H,? e t e SltUaU °K S were ^ m edified, using a software tool, to ensure that specific 
complexity factors were bemg experienced m order to validate our model appropriately 

O-igmally, we wanted to analyze Official Airline Guide (OAG) data to identify specWc situations 

walS madeT TS evaime du ? n S our simulations. Unfortunately, the necessary OAG data 

avSL^ltemahVe able U$ “ Ume ^ ^ Simulations ’ 50 recorded SAR data was S best 

Sh ° WS teSt matrix used for he validation simulations. Again for this table 
represents those scenarios presented under Current Procedures, H represents Half Free Flight ’ 

S^ a ", d * Free ™S ht P™*^ The nimSa 3 4 6 for 

conditions C 1 -6 for H conditions; and 1 - 3 for F conditions) represents the number nf the 
scenano (within each condition) presented to the Controller P bef ° f the 



Day 1 

Day 2 

Day 3 

Day 4 

Day 5 


Cl 

FI 

CG 

F2 

Cl 

Morning 

H2 

H3 

H3 

C3 

H4 

Sessions 


C6 

F3 

HI 

C6 



H6 

Cl 

H4 

HI 



C4 

H5 

C6 

FI 


H4 

H5 

C3 

H6 

H5 

Afternoon 

F3 

H4 

FI 

Cl 

C4 

Sessions 


Cl 

C4 

F3 

F2 



F2 

H6 

H2 

C3 



C3 

H2 

C4 

H3 


Table 9. Simulation Test Matrix 
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4.2.5. 3 Sessions 

TJ 16 station sessions lasted a total of five days, with two Air Traffic Specialists participating ner 

C f day SefV ^ 88 a P racUce da y to en s^e system operation and to verify P positiv e 8 P 
Controller response to the simulation environment. The ten (10) Air Traffic Specialists 

participating in the simulations were the same that participated in the Complexity Focus Groun 
AttirnMnn ST theS ° s ™ ulatlons - verbal protocol data was not collected during the simulation 
protocol datfl^ Wh SUCh r l S mtruslvene f or omissions may be encountered when collecting verbal 
eIchTn,S'fh^ hen pe h rforrmn « complex task s, operators typically do not speak aloud to explain 
tH n y m bave > 811(3 doin 8 so may prove difficult to do correctly. Thus the protocol 
• coUe ? ted after 3316 completion of the experiment. In addition, verbal pro’toco/analysis 

Derfomi! !?„? ^ n ’ ^wer-everythmg approach, but it can be used in conjunction with Y 

dl UStra ,c pomts> md il mfl y be hdpful in understanding subjects’ 

22“ ^? r Performance (Sanderson et al, 1989). Therefore, to minimize the infrusiveness of 

scenSio ’ ^ 00116011011 0f tlus *** was served for the debriefing session following each 
s f nano ' on f A** Traffic Specialist controlled the simulation (Radar Controller) while the 

Hlor S wh d ^ Pil0t ?° r ° ^ "StolSri FUU 

primary Pseudo-Pilot had full knowledge of these additional inputs so that he was able to^ 
communicate information about these flight path changes to the Controller, if necessary. 

HaVE ^ m0ming and afternoon session, the Controllers were presented with a 
T Hlght c . allbratl .on scenario. This scenario was rated to be of medium complex^ TTik 
call ? r f on scenario was intended to familiarize the Controllers with the specific sector de Jan tn 
us^ throughout the simulation, and to give them some 

Aflfrt,I a c Ct |^ at WCre ablC t0 ° hange mght P aths without acquiring Controller clearana 
prSd to condo" ttSSSr* *“ C ° n,r0l,ere Were giVe " 3 Sh0rt break and the " 

Before each test scenario. Controllers were presented with the flight strips for each aircraft in the 

sSn-flffi!? W6 n e aUowed equate the static traffic picture before assuming control This 

static traffic picture was combined with a quick position briefing, given by a Wyndemere 

researcher . highlighting amval aircraft and other specifics about the test scenario After thic 
position bnefing, the dynamic simulation was started. scenano. After this 

rating, with toe complexly measure 

awddiocurately represented a Controller's perception of air traffic ttontple^y Fo^owtoa tois , 
SCSSSi ° re ° f ^ reSearchera had h*" 10 «» notos dLtg toe 
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ControUer subject to the Pilots. This data was analyzed to determine if any operational errors 
occurred during the simulation, and to determine the level of efficiency of the flights in the 

n^w?°' LoWer < ^ Vels of^iency may be an indication that the Controller waT too busy to 
provide a more efficient flight, thereby providing an indication that the particular scenario^was 
overly complex. Similarly, operational errors may be considered as indications that the Controller 

ZeTf.? 0 b !T With ?* er tasks to ^ inflict or to resolve a 

Note that these ^ other tasks include all the tasks mvolved in the air traffic control process not 

!S?S the co ^ tlve tasl ^ s . Finally, the measurement of the number of commands that the 
r ^ Controller issued to aircraft during the test case was used as a measurement of the 
Controller s physical workload during the simulation test case. 

StSf? ft Controller .completed all 5 test scenarios, the two Controllers switched 

difWnt ft sessi °ns. The scenarios presented to the second Radar Controller were 
different than the scenarios presented to the first Radar Controller in the morning sessions 

had aJread ^ Seen , the morni ng session scenarios while actmg as a Radar 
• ' . the m 0rm ng sessions, the second Radar Controller completed the two calibration 

test scenarios to controi Again ’ 


4.2.5.4 Simulation Results 

w? ±e va f idati ° n simulations were used to provide us with further insight into 

Cnntnt? P^i 011 of ^ traffic complexity. The two major sources of data Ktod were 
Controller ratings of scenario complexity and time-stamped logs of Pilot inputs (which serve ns fln 

HehH a f 0n ° f the n y mber 1l of Controller-issued commands). Audio recordings from scenario 

rating? kh erc ifrtl > CoUected ’ t0 hel P us better understand the Controller-fssigned complexity 
rating Although these sources of data were combined to give us a better overall understand™ of 

* ^ o« “ch data sourS will be SSISJSS5wS5. #f 

C ontroller Complexity Ratings - Controller ratings of scenario complexity were collected after e«rh 
scenano using a paper-based, numerical rating scale. The rating scSrask^ snecd^5f 

L oler^pS^ of 

CMitfoller may feel that, tt,eoreticaUy!d!fo£ly toefv ^“shSt^ly'be 

3 S°£4„ ™“5 fore - m second Con^ m «y the JZtSSfi 

scenano a rating of 3. For illustrative purposes, assume that the “3” ratine is the lowest ratine 

Sn^dff 011 ^ 0 ^ ^ sl Z™ t0 ™y scenario. Although both Controllers assigned foe exactsame 
scenano different ratings, in both cases the assigned rating was the lowest ratinvTrroL .n 

SteS„« h , h ControU f r Wore ’ in order “ faflS 

«S5S5^2SoSSSa^^ - h c — (across -* 


SC ~.=(.SC M - RANGER 


RANGE 


+ 3 


value of 9 
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conditions. The confidence intervals 
adjust our complexity measure. 


were then used as a target range of values against which 


to 


Tune- Stamped Data U?gs, As part of our analysis of complexity, we evaluated the number of 
commands issued by Controllers under each level of complexity (see Figure 10 beW) 


Average Number of Commands Executed By Pseudo-Pil 
By Complexity Level and Condition 


low 


med 

Complexity Level 



high 


Figure 10. Number of Commands Issued For Each Complexity Level 

According to Controllers, the complexity of control is partly based on the number of aircraft that 
are present m a sector Furthermore, it is generally assumed to be Inm XTaTthe 

,he nun : ber of ' clMra "“ s «|UW also increases. We “v “age „um£r of aircraft 
Analysis of dm number of commands issued unSS^h SnpSxS y 

K^IO 20.07? ^“oof) ‘ SSUed Under thC higheS ‘ ‘ evel of ' «>">Plexity. as shown in 

rnimlv»rnf P^ticular result is not necessarily siuprising, we were intrigued by the relatively low 
number of clearances issued during high complexity, full free flight scenarios Our original 

aSSsr—sS.-. 
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Average Number of Commands 
Executed By Pseudo-Pilots, Across 
All Complexity Levels, By Conditl 
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Figure 11. Number of Commands Issued in Current vs. Free Flight Conditions 


Controllers did in fact issue more commands during free flight conditions 

Tnd ^ rtp^ely 26 ^^^ "T 8 090 

SIS’" 2000 f “V 0f VM1ical «P«r«ion S^ZTlSo) 

the Controller might issue two mmuadS^^ wthsepM-ation standards, and to ensure safety. 
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nrrlhtem^n : free flight, using a similar testing environment, will account for this 

problem by 1) having a better definition of the procedures to be used for free ^condition, and 
2) keeping a record of the observed clearances, to be compared with the time-stamped data loss Tt 
is worthy to note, however, that the reason that the free flight procedures were not^ore i 11 

defined m the current study is due in part to the fact that studying the physical communicE * 
workload levels associated with free flight was not a primary goal of f°n that 

” eW PrOCedUreS W ° uld have on,y confounded « ™—of 
4.2.5. 5 Simulation Comments 

heard comments throughout the simulation sessions was the fact that 
52*° experiencing a vety high level of complexity in his/hT^Tm™ t£e^ 

load o" both'SX^e*^^^ 

“ SySKm " “ *>““* by the 

The impact of weather on the complexity of control was also not evaluated Th^ nmieejo e 

E&, SiSSSn?' 

stoS,r d the of 
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5.1 ATC Complexity Measurement Environment (ACME) 

r wc h “ *» «*- » 

mode. Basic air traffic control data is raeived hv ri^ATPr-’ T U “w *” on ' lme > real-time 

SSSSSS^^SSrS 

™« -"Ode, arc intended io 

The modes tee CmG« S ?2L‘ * *» £"> <£■*» of air transportation. 

of these procedures have been described in Section 4 2 5 I so cmlvtiS” exacl defmitions 

the ACME system itself will be described here. , so only the use of these modes within 

5.1 .1 Flight Path Modeling 

%£ PtaSd h TrliS“ m ^: radar " 1 & a " d ^play computer is 
be received from the primary ATC dla ' FI '8 ht Pb " ^formation will 

controlled region of SmS aUb2SS?« 52? \ VT* that wU1 flying through a 

Aircraft that fre flyi^Sd? ^ "“f™ SyS * m ' 
Controller’s sector without havTnea Kt 5S? ? y m “"I** ** « allocated to a 

computer system. sSISaSRK %.?* ^ ^ ^ ^ Primary ATC 

CoSSs ■*» ^ex^Tthe 

sector airspace if the VFR aircraft may be^me ffactor ' SSK? d “ “* ° f her 

Ss la ScaS 1 the roSrf r ° Ute ° f mght ^ the aircraft 

through its full route of flight to touchdtwn ^ current position of the aircraft, 

used from the Flight PtaX KZ*?,q^ .T? » daahnadon. The information that is 
altitude. In the ACM2 o£S S “ d fd «i 
aircraft. During the Free Flight mSs SSL IE Rou,es “« assigned to each 

flight from the aircraft’s current ^onrcT’tetoti^ ^ *« "»*■ <* 

T~1 .1 - 


modeling sy^^ EESMjS 8 " 

the generation of wind-optimized routes Both of thL* j— 86 ^^pa^ (SUA), nor does it include 
system would have increased the arci^acv of theo^c^T? 1 ? 5 the mght P ath modeling 

ss^a-i ^ k-j— s* 
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sjMSsrjssr as ‘KSfisri" 

already exist with very accurate flight path modeling systems such a* CTA<z ;* that 

ZhSSST 1 sisnmcenl effort “ development of 

5.1 .2 Trajectory Synthesis 

°L W ? r l° n **£ *9® system ’ 311 effort was ™de to utilize the Trajectory 
° rS) fr ° m the Cent er/TRACON Automation System (CTAS) The £ effort 

undertaken was to integrate the TS from the Build 1 CTAS system because toe TS hi 

converted into a library with a simple function call interface. However it was found that toil 

pS„l A S^ 

However, it was decided that the increased effort required to interface with thic t<? , ' , • , 

est interest of the completion of the work outlined in the Statement of Work for this Itudy. ^ 6 

ssm mmmm- 

to? ACME SS, f ° f night f ? * n individuaI " CTaft - » Rectory 
CTAS TS module have been maintained in the ACME TS module E wm X^e^« 

5.1 .3 Complexity Analysis Tool (CAT ) 

tSi KtiS Vi EcA 8 T h m^ [“nr “ ACME “oOule called the 
to determine which sector the aircraft will be in at the prediction tae ^ ^ yZCd 

** r T 0nS “f I-,' to>ke up ATC 

ss3S?£s saiaisrsraii. 
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evaluation time. For example, one of the complexity components is a count of the number of 
aircraft that are either climbing or descending in the sector (CoD). The CAT module computes this 
complexity component starting at the current time and looking forward into the future bv a time 
parameter that can be varied by the user. The complexity components are calculated at time 

7* the future b y a parameter. For example, the complexity components may 

be computed every minute for 120 minutes. This gives a two hour look-ahead for complexity 

the complexity component that measures the number of climbing or descending 
aircraft, the CAT module searches the list of track points for each aircraft for each complexity 
computation time interval and checks to see if the track point is within one of the sectors for which 

nrZl”? 18 7 e ^ Ured P° int is within sector, and that track point is in a climbing 

or descending flight segment, then the CoD complexity component will be increased by one unit 8 
for that time period in that sector. y 

However, the Air Traffic Controller is constantly predicting situations that may occur in the future 
and formulatmg a plan to deal with such situations. Thus, it is often future events that affect the ’ 
overall complexity of the situation for the Controller at an instant in time. For example if an 
aircraft is going to descend from cruise to approach its destination airport, the Controller must look 
forward m tune and predict the possible interactions between the tending aS and Sr 
aircraft that the Controller is responsible for, or will be responsible for, in the future The 

STa dSZg S" “ CUrren ‘ by ' venB wU1 ««»«. in Urn 

pnm n/Yn*»nf atl ^ n USed f f ° r . the . Dm *£ T Complexity Measure is to measure the individual complexity 
components at an instant in time. Then, the effect of a given complexity measure on the overall 

S Sh 1 * u 5 U bC ® fun u ction °( t* 1 ® component complexity at time t(n) through t(n+m) 
Here, m is the look-ahead time for that particular component complexity measure. 

5.1.4 Individual Complexity Measures 

SfifSSSf 8 f hs d ?™ ht 7 e differen t complexity components that are computed by 
the ACME system and used in the Complexity Measure. Following the description of the 

complexity components, the methods that are used to formulate the overall complexity measure 
CAT m e od C ule P ° nent wUI * described ' of these activities are 

l^^ S \T P ^^ 0 t mputing 5 e f om P lexit y algorithm was to examine the complexity factors 
mdmdually, to determine whether it was feasible to mclude each factor in the algorithm Ideally 
wewould have liked to include every factor identified in the Focus Group Sessions in our 
complexityalgorithm . However, for some of the factors (CRD, FAC, SEP, and WND) we do 

not have sufficient data or resources and would therefore not be able to properly validate these 
measures during our simulations. F y cse 

C *P and FA C were not used in our algorithm is primarily due to our limited simulation 
capabilities. The impact that coordination has on the complexity of control can only be truly * 
investigated with a relatively large number of Controllers and Pseudo-pilots. The intricacies of the 

n o ^^ARTrr°fr?>' rS , mUS1 "f 1 ” 0 Wi,h 0ther (eitter within their oZ^TO 

fi 1 f ? dlt1 ^ ^ only 1x5 if there are a number of other people available to 

fill these additional roles. The impact of separation requirements (SEP) on complexity was 

considered in this study is because we wanted to focus our study on the elements of the traffic 
situation that impact the complexity of control. Separation requirements (i e miles-in-trail) are 
nnpc«ed by an Air Traffic Control facility and are therefore not considered to be element of toe 
ltSef ‘ ^ mCC system capabilities do not allow us to simulate flow control ^ 

8 mpl,a SEPhlSonthe complexity of control could no, be 
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y md and weather information is not available for simulation at this time Therefore w, 

tZuUt examme ^ mpact ** WND ’ WDN ’ or WST has on the complexity of confrol For th, 
simulation purposes, we assumed zero-wind, clear weather conditions across all scenarios. ? * 

B^ed on the data collected throughout the study, the individual complexity factors along with „ 
descnption of how they were initially computed, are presented below; Y ’ S * 

5. 1 .4. 1 Aircraft Count (ACT) 

This complexity component is simply a count of the number of aircraft within the lateral on H 
altitude boundaries of the sector at an instant in time TheAiro-aft CW Vf d , 

K u«d to provide an indication of the number of deaiMt^t^teJSdrfl SZn'"' 5 ' 
and the number of utdividual aircraft entities that the Controller has to meZtyLnitor Zd back. 

5. 1.4.2 Convergence Angle (ANG) 

mmmmrnm. 

mmmwm ^ 

5. 1.4.3 Crossing Altitude Profiles (CAP) 

tfcoT rr wm 

oZE"e“ aintations in which the ConZlertt e=etS ^&-, rel y 
5 1 .4.4 Climbing or Descending Aircraft (CoD) 

to address is increased through the interactions of flints l Itu^Tveb ^"h^gtog^" ““ 

5. 1 .4.5 Closest Points of Approach (CPA) 

^rn P of^h™,t7i^^ Z£g£ZZZZ ' Z * “ 

predicted instant tu 11c , fU- _ i . ^ again that this instant in time may be a 

However the complex!* comp^riS^Z-ZZd^ZS f Se P ara ‘ ion ' 

“■ rather tha " states 
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nation of a predicted separation that would cause action on the part of the Controller 
In the computation of the complexity measure, one unit is added to the CAP comnnnont at f ; er ' 
.. wh.ch two aircraft arc predated to be wiftto 8 miles of e“er ^ J TS2 
used as an indication of a predicted separation that would cause heightened separation monitori™ 
between the two aircraft by the Controller. One half unit is added to the CAP component at anv 5 
tune at which two aircraft are predicted to be more than 8 miles apart, but less than 13. y 

5. 1.4.6 Aircraft Density (DNS) 

^n^lii? nSity iS **, airCraf i 00X1111 divided hy Ae usable 8111011111 of sector airspace This 

amoL sspzi'gss on ^ 

5. 1.4 .7 Intent Knowledge (INT) 

111501111811011 aboo t the intent of the aircraft is also evaluated. This measure is 
somewhat simplistic at the current time, being classified into three levels of intent knowledge 

destination, altiS^KTti” J** 

5. 1.4.8 Aircraft Neighboring Conflict (NBR) 

sras irrs“T 0 £*t^is’ t ■ 

fa? °” e U ” it fM ^ * Within 10 “Ues and 2000 vertical 

5. 1.4.9 Conflict Near Sector Boundary (PRX-C) 

complexity may be increased by having to coordinate with adjacent sectors to complete JJ? 01 er s 
5.1.4.10 Altitude Variation (VAA) 

This component is a measure of the variability of altitude of aII of th* , L 

instant. This complexity component is computed by the ACME system but ? any 

from the Controller interviews that altitude variation^ itself has Miy^^wct^on Comrollw complexity. 
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5. 1 .4. 1 1 Heading Variation (VDF) 

TTiis component is a measure of the variability of heading of all of the aircraft in the sector at a time 
mstant. A higher heading variability of the traffic situation provides less organization of the traffic 
flow for the Controller. With lower heading variability, often the Controller can group individual 
aircraft together as traveling in the same basic direction across his or her sector. In this manner a 
mental dependency between two aircraft that provides a quick check for separation can be created 
This simplifies the situation for the Controller. 

Since heading is a cyclic parameter (0 to 360 is the same as 360 to 720), the variation is computed 
through the use of a pair-wise minimum heading difference squared and summed over all aircraft 
pairs. Consider, for example, the standard variation calculation for one aircraft heading 359 
degrees and a second aircraft heading 001 degrees. A standard calculation of the variation of this 
data set would use an average of 180, and square and sum the differences between the headings of 

touse S^rca^ 6 aVCrage heading of 180 ‘ However, a heading of 360 is a more correct average 

In cases with more than two aircraft, the best average to use is not as readily apparent. The 
variation calculation used for this complexity component removes the use of an average value and 
only uses differences between two headings. In this manner, the numerical ambiguity that results 
from the standard variation calculation is avoided- 8 y resmts 


1 


(«)•(« + !) 


LLv, J M & ~ hd sj ) 2 


5.1.4.12 Speed Mix (V AS) 

™"^P°n? n U S a mea f L l r ! ° f , the vari J abiHt >' of ground ^ of all of the aircraft in the sector at 
T 1116 Vanablllty , of g rou , nd speed affects the complexity of the traffic situation for the 
Controller by causing potential overtake situations, and increasing the difficulty of predicting 
relative future positions of aircraft because of the differing ground speeds. Since ground speed is 
not a cyclic parameter, a standard variation calculation is used. 

5.1.4.13 Aircraft Proximity to Sector Boundary (PRX) 

This complexity component is a count of the aircraft that are within a threshold distance of a sector 
boundary at a given time mstant. When aircraft are near a sector boundary, a greater amount S 
coordination and monitoring is required, which can increase Controller complexity. 

5.1.4.14 Airspace Structure (STR) 

This complexity component measures the conformance of the traffic flow through a sector to the 
geometry of the sector. In general, sectors are designed for specific air traffic flows ^or 
example, arrival sectors are generally designed to be longer and narrower than normal sectors and 
are oriented toward the arrival terminal area. A large percentage of the aircraft that fly through this 

SS offiSc^ amVal “ “ - S ° a* ^ ^ dirSZ^, 

A Controller’s complexity can be increased if there are aircraft flying ‘against the grain’ of the 

SI?' J 0 <Urcraft m fl y“« across major flow of traffic and/offlying across 

of If • ^ C°mroller must engage in additional oonmamoJtodngSor 

coordination. The computation of this complexity component is performed by calculating a mai or 
was and aspect ratio for the sector. Then, the difference in heading between Lch SrcraffidT 

major axis is computed. The squared deviation from the major axis of the sector is weighted bv the 
aspect ratio and then summed over all aircraft. weighted by the 
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5.1.5 Factor Combinations 

The next step in fomulating our complexity algorithm was to determine how the individual factor 
weightings might be changed when combined with other factors. To begin this process we first 
examined the absolute ratings given to each factor pair. We used this daS to get S 
understanding of the absolute level of complexity associated with two combined factors Tables 
containing this data are presented in Appendix A, along with the Factor Intervkw SrsummirL. 

variances. For all statistically significant results (i.e., the two distributions were found^be 

nSih^/ dlff f rent) u t : tes .l results Presented and discussed in Appendix A In addition 
hSnn fh f0r Why ^S^ant difference was found, and the implications that ttefindK 
Sr men ‘ of ^mplexity will be given. An example shoeing this pr^ssls g 


Single Factor Ratings 


CAP 


Factor Combination 
Ratings 

CAP x STR 
CAP x SUA 
CAP x PRX 


8 8 


5 

3 
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8 
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9 

9 
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S4 
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S6 

S7 

S8 

S9 
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6 
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6 
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2.044 

S4 

S 5 

S6 

S7 
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S9 

S10 
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1 o < - ,) 

6 

5 

3 

7 

7 

5 

1 

5.6 

i 2.3664 

2 

5 

4 

9 

4 

5 

1 

4.8 

2.6583 

8 

6 

3 

9 

6 

5 

9 

7.3 

2.2136 


In this example , statistical analyses revealed a significant difference between the 

SUAio- a ??f a5Slg nni{ 0r 1 CAP l and the absolute stings assigned for [CAP x 
SUA] (t = 2.26; p < 0.037, two-tailed). However, we see that the absolute ratine 
of complexity associated with the combination of these two factors is significantly 

a£Z k T thea Z S ° lUte ra ? ng °f com P lexit y ^signed to the individual factor [CAPl 
acme A possible reason for this might be explained by the fact that SUAs are 
usually not located in the direct path of a portion of airspace If this shnulH hnnn 
however, the Controller might op, to LI for eLplLLlIung Lcra^go 

°deof h[sUA m<t f desc f^"« ^craft go amundthesoMh 

tne f UA - Therefore, the complexity of that scenario would not be 
considered as great an impact on complexity as the presence of a large number of 
crossing altitude profiles between two air cr eft. * 

An interesting problem is highlighted in this example. From an intuitive standnoint 1#1 

ssaAs^sSSSsSSs^SSl 
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qualitative, interview data, the Controllers may have inadvertently assigned a ran kino 
assumed (but unknown to us) relationship between those Zo factom g *** ° n “ 

25^“*“ { U f ther evi denced upon examination of a simple correlation matrix between individual 
factor, absolute ratings. In many cases, the correlations obtained do not provide anv reliable 

SESSSSn* ,' he COmbined fac,0r “** ■ AJtho “* h "O - W-l tatstiuate 

ZX ; E Curs , or y examination of the data does indicate that Controllers may h^ve § 

rimkhios °L addltlonal relationships between factors when assigning the Combined 

JS * hJ n identifying weightings for combined factors, we decided to use the more 

detailed-but difficult to quantify due to its variance and subjectivity-interview data as a hack f nr 
ur assignments. We believe that through a careful examination of the interview data we will he 
better able to capture the essence of the Controller-assigned weightings Due to the nature n/!h? 
mterview data^ we also realize that a certain level of researcher ££L, 

weightings. However, smce this study is designed to allow us to further modifv ™ 

we feel *“ “• «*” - 

5.1.6 Overall Complexity 

=S$K^ESSS23S33r 

study that formulations other than additive mav have been annrnnriate . 1 1 16 

cases that indicated a different form of ctaabiB^hSaE ^^ 

nwe appropriate, there was a clear operational combination of factor* “ S Sf i7wt 

Three different approaches are used to combine the effects of the individual comoleYitv 
components into the overall Complexity Measure. These approaches are used ufmodd the 
p ational impact that each of the complexity components has on the overall comolexitv Th * 

s.chS'mS'i be «» 

Complexity com P 1 “ t y « then combined additively to form the overall 


5. 1.6.1 Maximum 

SEZSSfgE *! » whUe 
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O l =W i MAX(F i (n),...,F i (n + m)) 


5. 1.6. 2 Cumulative 


Other individual complexity components model events that are uredicted to occur i„ etc 


N+m 


n 


5. 1.6.3 Weighted 

ia simile to the Cumulative function, with the addition of a tofbaSSfe^ac™ 5 aPPr ° aCh 

n+m 

°i = W i '£(n+ m -k)F i (k) 

n 

^&&^^ssxsssasr 

number of factors that were included in thp mnr i/»i tu SS was ver Y com phcated due to the large 

itself. In summary the vSon me&Solo^lS * complexities of the simulation test matrix 
generate a numSo^S"SS 1 ^^ ,Kea S. m, ‘ , ^ 1 ">«*>« <o 

controUed in simuladon by thVsm^Ai Traffic f y ’ ?“* of these scenarios 

Sessions. The results from Z ifcl'S “ J>e Complexly Focus Group 

™; Belo »- « scribe to iterative approach we mote 


5.2 Refining The Complexity Measure 

5.2.1 Measurement Iterations 

■** t. 1 ° refme 11,6 

Indeed, given the subjective nature of the feta we ww SS^sTS 8 SUTlulations - 

process of foimulating a mathematical model of Ln^iwT^ t ls not sur Pnsing that the 

the simulations, each simulation SS “ iterative one - to 
complexity values for each individual factor as wHI oc ~ resulting in a set of 

data was not shown to the cThXl Sg ft^“M o r^ mPl ' X . 1 S' ™? ““P^ty 

sr y - “ s were abie - it 
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In general, the methodology used was to refine the ACME system so that it could accurately 

hv^T W? t0 C T tr ° U u r perceptions) toe complexity of an air traffic situation on a factor, 
by-factor basis. Having done this, we then analyzed the Controller ratings from the simulation 

tSTndfvid ^TT* th H?J atl0nSh If CXiStS between toe ratings thatControllers assigned to 

the individual factors and the overall complexity ratings. In doing so, we are now able to 

individually analyze the impact that a specific complexity factor has on an air traffic situation as 
well as understand how that individual factor contributes to the overall complexity. 


5. 2.1.1 Individual Factors 

St6 P !^ en t0 repin ® tbe ^dividual factor complexity measures was to plot the Controller 
ratings against the initial values computed by the original complexity algorithm, for each 

complexity factor Each simulation scenario was run through the ACME system resultine in a 
complexity value for each factor, for each scenario. These values were a mathematical * 
representation of the information collected during the Focus Group Interviews scaled to ranee 
from a value of 1 to 10. Plotting this information allowed us to see how our initial computations of 
AppenTiTc) eX y C ° mpared t0 Controller ratin « s taken during the simulation sessions (see 

AoIro Sing f a Sim F le Unear regression equation, we analyzed the Controller ratings versus the 
ACME system ratings to generate a new set of coefficients to be used for our cSe^v 
algorithm. The reason this iteration was conducted was to try to match as closely as possible the 
dgontiim values with the Controller ratings. Again, the situation sceA^os wtreLTiugh Ae 
ys em (using the new coefficients) and the complexity values were recorded These 
values were also plotted against Controller simulation ratings ^allowing TtoS hSv welTour 

!n Hiv'd C T P , lty dg r rithm was computin g values that matched controller perceptions of the 

C -? m ?r Xlty faCt ? rS 0nly ’ ^ example results table (for the density P [DNS] factor) from 
these individual factor analyses is shown below in Table 10. The corresponding data D lot is 
shown in Figure 12. The data plots for each complexity factor is found m Appendix C. 


Regression 

Statistics 

Multiple R 

0.67244958 

R Square 

0.45218844 

Adjusted R 

0.36885511 

Square 


Standard 

1.13499843 

Error 


Observations 

1 3 



Coefficients 

Standard 

Error 

t Stat P- value 

Intercept 

CONTRIB 

0 

5.57890637 

#N/A 

0.28336713 

#N/A #N/A 

19.6879096 1.6758E-10 


Table 10. Example Regression Result From Individual Factor Analyses 
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Figure 12 Example Data Plot From Individual Factor Analyses 


notiww r Uie ^ from our complexity algonthm (on a factor by factor basis) may 
match . Controller ratings, it is important to note that both the complexity algorithm Y 
computations and the Controller ratings tend to follow similar trends across conditions (C H or 
F) and complexity levels (low, medium, and high). This suggests that our complexity measure is 

XZITa S-by-hcSbaS" 5 ’ ’’"“P* 0 " ° f h ° W ■" different traffic 


5.2. 1.2 Overall Complexity 

OUr c , om .P lexit y algorithm was to determine how the ControUers combine 
e efforts of individual^ complexity factors into a rating of overall complexity. During the Focus 
Oroup^temews, the Controllers found it very difficult to explicitly state the relationship that 
exists between individual factor complexities and the overall Complexity of an S^ScriSon. 
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g#^^r 


Regression Statistics! 

Multiple R 

0.87668129 

R Square 

0.76857008 

Adjusted R 

0.7171827 

Square 

Standard 

1 .06298846 

Error 

Observations 

81 


Factor 

Coefficient 

Intercept 

0 

INT 

0.05121707 

DNS 

0.46770975 

ACT 

0.02455453 

CCD 

0.16164556 

NBR 

0.06071066 

CPA 

0.07104819 

AN3 

0.15247093 

PRX 

-0.1406604 

PRX-C 

0.10745819 

VAS 

-0.0045182 

VDF 

0.10182903 

STR 

0.09646242 


Table 11. Coefficients For Computing Relationship Between Individual Factors and Overall 


Note that in the above coefficients two of the valuer rpp y,^uao\ _ 

^ssfSS?® 5 ^ 

in the overall complexity of a situation with a Si 
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indicate that that there should in fact be a positive correlation between the PRX factor and the 
overall complexity measurement. What this suggests is that there are unknown relationships 
between other factors that we have not yet been able to account for. 

Second, the amount of data we had available to perform the regression was most likely 
sutaantmUy lower than would be needed to get a statistically significant result. In fact, it was very 
difficult to assess the complexity ratings for each scenario across Controllers because each ^ 

workeda subs r et 0f tbe toUd number of scenarios. Because we wanted to examine 
ControUer 8 perceptions of complexity when placed in a monitoring role, it was necessary to 
ensure that the Air Tr^Fic Specialists were not presented with the exact same traffic scenarte when 
assuming the roles of both Radar Controller and Radar Monitor. In addition, the Controllers we 
used in this study were all full-time, active FPL Controllers. As such, they were not able to 

M^mr? 1 fHflT 0Un | t H 0 h least 1 fuU da y as a Radar ControUer and 1 full day as a Radar 

Momtor) that would have been necessai 7 to completely run all 13 scenarios. 

After dropping the negative coefficients, the complexity measure was analyzed with recorded 
affic operations to evaluate the realism of the complexity measure in actual traffic operations, 
ough these analyses, differences between the simulation scenarios that were used with the 
controller subjects, and actual traffic operations were observed. Many of these differences were 
also noted by the controller subjects. One of the primary differences is that the simulation 
^ I \ a f los w ^ e specfficaUy developed to provide a single traffic situation for the simulation subject 
to address. However, actual traffic operations are more continuous, with aircraft entering and J 

rtf T ? r at mUC u m ° re dlverse times ’ Frances resulted in the need to modify 
f ^ or contributions to the overaU complexity. The final complexity measure is shown 

is sliown in F^ure 13 of fulal com Plexity measure to the overall Controller ratings 


0.0172 x ACT (MAX., 10.0) 
0.328 x DNS (MAX., 10.0) 
0.0498 x CPA (SUM, 15.0) 
0.1070 x ANG (SUM, 15.0) 
0.0426 x NBR (SUM, 15.0) 
0.0754 x PRX-C (SUM, 15.0) 
0.1134 x CoD (SUM, 15.0) 
0.0709 x VDF (MAX., 10.0) 
0.0 x VAS (MAX., 10.0) 
0.2 x PRX (SUM, 10.0) 
0.0676 x STR (MAX., 10.0) 

+ 0.2564 x INT (MAX., 10,0) 
OVERALL COMPLEXITY 

Figure 13. The Final Complexity Algorithm 
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6.0 


ANALYSIS OF TRAFFIC COMPLEXITY 


6.1 Rationale 

^i^Zi f ? CU8 °! Study has been V se of 1116 complexity measure that has been generated and 
v idated to conduct a comparison of similar traffic situations under cunrent procedures and free 

c.tl^ UKS ° f ? er Stud i es hav . e performed similar analyses of traffic under the free flight 
a .£ / owever these studies only considered changes in airspace density and conflict events 
nnrtr^f .u fouri f d ?L ou Sh controller mterviews and simulations that have been conducted as 
E aS study density and conflict events do play a major role in the overall complexity 
of the ATC situation, but these two factors do not determine the ATC complexity exclusively ^ 
5“^ ““2 ot 5, er characteristics of the ATC situation that will be significantly differenfmder 
e identified free flight procedures. These characteristics must be considered in any analysis that 
tbe complexity of the ATC situation between clearance-based procedures mid free flight 
This study has attempted to consider all of the major characteristics of the traffic situation in a ^ 
comparison of current procedures and Free Flight procedures, as will be described below. 

6.1.1 Caveats of the Traffic Complexity Analysis 

SShIS? OU f analySlS °/ f 8 ® 0 situations was conducted with a fast-time simulation system 
. atd £f " ot consider some of the important elements of a truly realistic traffic situation For 

dynamic wul ds, and hazardous weather are not considered in this 
t Y' - f ona y* seme components of the fast-time simulation system have not been develoDed 
of accuracy that matches the current state-of-the-art. Examples in this categ^ areThe 

nS? 1 7 synthesis lo 8 lc > and the adaptation data that specifies the ATC Preferred Routes that are 
used under current procedures. Initially, an attempt was made to use the trajSSy svnthesL 

we^oted m S CTAS B " T? TJ* aCCUrate H °wever, a nLiber of bugs 
were noted m the CTAS Build 1 TS module, and the TS m the NASA baseline did not have a * 

convenient software interface, due to the use of shared memory. 

SghtDatTcMte?f^c?d?tf ^ A WaS ^ ? r the Study was obtained fr om the FAA’s National 
the S F A A Th^ " (NFDC ] data : A number of airspace data files are maintained by this group in 

the FAA. These airspace data files include data for Navaids, Fixes, Airways Airports and 

gSEFE* S® 1 Use Airspace, ATC Preferred Routes! Others. The 
N iH F' fde ^ the AT 9. Preferred R °utes data makes reference to data contained in the 

F if es - Airw ays, Airports, and SIDs and STARs files. Unfortunately it was found that 
SSLl ? f E es d ° n ? f have consistency between themselves. The DIRECT research temn made 
ive e orts to force consistency into the data but this was not possible In addition ronfa^f 
was made with the organization of the FAA that maintains the NFDC data to deteS there wa, 

Se ^C^T^^paT n ° t i e l eaSed With the NFDC data that would make it consent 
currentlv ° f ^ Stated 11x31 * e data is not consistent, and in fact, little effort is 

, • e „ ntly mad ® to ensu re such consistency. The system that maintains the NFDC data is currentlv 
being upgraded by the FAA, and one of the major goals of this upgrade is to 

consistent date^ould be ob£d ^^eShS’dTte ‘LhfromnTety^ 
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was to use a direct route of flight in place of unresolvable flieht 
current procedures. 


segments, even in the model of 


6.2 Analysis Methodology 

procedures ^ “* COnd “ cted ^ dearance-based 

SStirfSif h** if®*” ? e [ f ° r T ed by the mTRE corporation, in support of the FAA The 

^^■=335SSsSS^ i *ss' 

considerabon of all major baffle chtbacteristics, rather than jostdensi^r tSicUv^b. 

AKTCr“p«Sr“ thal us ? d for this anal > ,sis were deriv « 1 from (SAR) data from Denver 

AKTCC. Efforts were made to obtain OAG scheduled flieht data, but , Denver 

TteaSsrussess 

date was used to generate a full flight Taj^y £ ^ l ample 

“ d anaiyze «»”piSSy m of^e been 

5£S 

TRACON sectors. ^ ATC complexities were not analyzed in 

^mplexity of the ATC traffic situation was analyzed for all sectors in Denver ARTrr > • 

data describes the SrniW AeaSat sectors ™fd« S *“ ^ "f 00 " 1 of «* ACES 
known, the ACME system applieslhe Complex^ mSJ ATC .T^ is 

SSSi'Sgg?' COmp °" e " B ’ “ d - tHen 

** SAH d., ™,pfc f„ ^ ful . 
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free flight simulation run. The fir st goa l of this comparison is to attempt to validate the results of 
this study with the results of the MITRE study previously referenced. Since the two studies both 
attempt to evaluate the changes in density and proximity events under free flight, we exDected to 
obtain similar results. The second goal of this comparison is to determine the change in overall 
ATC complexity between current procedures and free flight. Thus, three different complexities are 
evaluated in the comparison. The Density and Closest Approach complexity components are used 
to compare against the results of the MITRE study in changes in density and proximity events fr 
is important to note that the third type of com plexi ty that is evaluated in this study is the overall 
complexity, which was not examined in the MITRE study. 

6.3 Traffic Complexity Analyses 

6.3.1 Current Procedures Vs. Free Flight 

The density and the number of closest points of approach associated with current procedures and 
free flight were compared on a sector-by-sector basis across 15 minute time intervals (for these 
C VhT 00 *' thC TRAC , 0N was considered to be one sector). A count was made of all cases in 
W k- CU ^ nt P rooed F es com Plexity was higher and cases in which the free flight complexity 
was higher. The results from a 6 hour System Analysis Recording (SAR) data sample are shown 

below. These results are accumulated over sectors and time intervals. Note that these results mav 

chLige'were ignored 6 ““ °* “*** measurements because 08568 in which the complexity does not 



CP > FF 

FF > CP 

DNS 

293 

274 

CPA 

344 

338 


Table 12. DNS and CP A Comparison - Current vs. Free Flight 


lQ0^ nO R e ^ at H^- milai j reSUlt u S obtained here as in the previously mentioned study (Ball, et al 
® th smdies indicate that the free flight procedures decrease sector density and conflict’ 
events as compared to current procedures. y 

A comparison of the overall complexity between current procedures and free flight was also 
conducted. These results, again accumulated over sector and time intervals, are presented below. 



CP > FF 

FF> CP 

Overall 

145 

971 


Table 13. Overall Complexity Comparison - Current vs. Free Flight 

a rr f *“ ? 0mparis0n mdic&tt a substantial increase in the number of cases in which the 
e all ATC complexity is greater under free flight than under current procedures. Note that many 
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more cases are involved m the comparison of overall complexities than in the comparisons of 
density and conflict events. This is caused by the fact that many more traffic characteristics are 
considered m the overall complexity measure, which results in fewer situations in which the 
complexity remains the same between current procedures and free flight. 

This result provides a very strong indication that measures of density and conflict events are not 
sufficiently representative of the overall complexity of ATC. The differences in traffic 
charatensflcs - other than density and conflict events - have a significant impact on the difference 
in overall complexity between current procedures and free flight. 

6.3.2 Complexity Dependencies Between Sectors 

tr ^ 1< ; com plexity analysis plan, we intended to examine the potential for dynamic 
• P 6 f^Hpies jmd rekh<mshg>s of complexity factors between adjoining sectors However after 
interviewing the Air Traffic Specialists, we realized that this analysis would not be meaningful to 
perform. Controllers stated that the complexity of an air traffic station de^ tZTtoSt 

fiT thilw^n ° r I 'V rther ’. they stat r ed that at any given time, they could be extremely busy 
the -ro*r d reciuir ® th ® assistance of both a data-side and an additional radar Controller) but 
t e Controller responsible for the adjoining sector would not be busy at all. These large 

differences m workload and/or complexity between adjoining sectors e a direct result of the 
characteristics of the traffic flows through the ARTCC airspace. 

f^ft lfalarge nUmber 0f e aat-bound jets are simultaneously passing through high-altitude 

the of control due to aircraft densfty (of simply 

the number of aircraft) might migrate across sectors, to sector “B ” as the traffic moves eastw P flrd 

mi & &hon of complexity from “A” to “B” would only occur ifthe aircraft were df 
going to the same destination. If as is normally the case the airfraft or** fkrinrr _ 

S desdnatos (b° th m the Nor* aad the paths w”S * 

quickly diverge. Also, the potential conflicts experienced in sector “A” (e.g* the east-bound traffic 
SS“ Sing r h an S al traffi< ? would most likely not cccm- in sector “B” because of the difference in 
TeS “A^Put 6Xlt w eXP f K CnCed m SCCt0r “ B ” would be ^ reatly different from that of 

complexity in ^ “ de ' e ™ ine the 

6.3.3 OAG Complexity 

TTiroughout the course of the contract, Wyndemere tried a number of times to obtain valid OAG 
data for use in our analyses, as the OAG data would provide a more complete anatai? of Z 

CenSd £ Ser ARTCC SSS*' ^ ^ contac ^. both «* VOLPE Transportation 

*?,??* was com P letely corrupt and subsequent attempts to contact VOIJPEfofa 
replacement data set went unanswered The Denver ARTPP FariiiK/ ♦ u u j. 

£? f ctxrect hardware cordon 

equiptnit iSmpMfte ™ "° d ” Kly Way *° COn ’P lete FAA P'ocuremem of the ’ 
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7.0 


COMPLEXITY REDUCTION AND THE DIRECT CONCEPT 


7.1 Complexity Reduction Heuristics 

In addition to measuring complexity, as part of this study we also examined various methods for 

e^h X! 1 based 0n * e characteristics of the identified complexity factors For 

idmtmif^TW 1 S nt ^ C h ° mp0n f ntS ’ one or more heuristic complexity reduction methods were 
tiffed. These heuristics apply to potential flight path modifications that could be aoDlied to 

to *? uce . the of the ^ baffle situation far the Controller. The DIRECT System 

wrinn h heunstlcs Wlthm an iterative algorithm to reduce the complexity of the air traffic 
situation by reassigning sector airspace for the Controller. y tlc 

It is important to note that the following complexity reduction heuristics are not intended m • a 

conflict resoiution. Rather, the complexity reduction heuristics should provide some level of ™ 6 

additional organization to the traffic flow, such that the Controller is better able to perform the 
Drooe« resolution task. In addition, the complexity reduction heuristic^eTtencffto™p^rt the 
5 Dynami< r Refectomation. Toward this end, the complexity reduction iSriSffSSuU 
also result m some level of dehneation between major traffic flows such that different SSn^ 
could be created to deal with independent traffic flows. Sectors 

^so, as described in the Operational Procedures section below, flight path modifications will rmiv 

t0 l m ° rde u t0 081186 ^ aircraft t0 en ter a different sector than the one that it Y 
? ave u se Wlth0ut the flight path modification. Thus, the only way that a 
, P reduction heuristic will have an effect is if it is partially responsible for an aircraft h<-ino 

f** N ° te tha ‘ the mgh ' P ath and sector ainspaS 

/h be . an iterative process, so that moving an aircraft to a different sector may not 
necessarily be a significant move, if in fact the sector boundary is also being moved toward th^ 
subject aircraft. Although the generation of the new flight pMh td “a SpXmoSon 

foraircraft “rnf StUdy ’ “ ch ° f the heuristics "a 11 result in a unique flight path 

will be te suTec^ fure^k SUg8Kted flight P ‘“ h modifications tat ° a «&< p P ath 

? v “ WU1 g Subs f Ctions ’ each complexity factor is identified, briefly defined and is followed 

^T^l^ e th° mP l e !l lty reductlon heuristics. Some of the complexity reduction heuristics haw 
been analyzed through the creation of scenarios that demonstrate the effectiveness of the henrictir 

!fr^rc Cm ^ tbe ^P 1 ^ hie air traffic situation. The analysis of these scenarios through th<» 

he^tic. yStem Wm ^ deSCnbed d680ri P ti0n of hie ** 

7.1.1 Aircraft Count (ACT) 

b6U8 ^. t0 hie complexity associated with the number of aircraft woddbe to forcesom^ 
aircraft to avoid the sector in question altogether. This sector avoidance hewritS 
implemented far enough ahead of time so that the number of aircraft in any given sector d^e< nnt 
exceed some capacity limit - defined by other sector and traffic characteristic. d 

7.1.2 Convergence Angle ( ANG ) 

“therateSc^f “'1?^ ° f T h siluatio " based « «* 

flight paths thatresult in high^compleM^ TOn^cf^omehies sodutt^ fl^hfpafh confhrt^ ^ 
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geometries will result in lower complexity. The benefit of using this heuristic is that in some 

nflthil?^ S V the t^ 0mp CXlty confli< ? t wlU sun P 1 >' be reduced, but at other times, such a flight 
path modification may result m a complete avoidance of the conflict. 

7.1.3 Crossing Altitude Profiles (CAP) 

h?Hi C ^ PleXit J C ° mp0ne " t ls .^, c l ount of ^ number of pairs of aircraft in which one aircraft will 
S mbu ;f a ; d one aircraft will be descending through the same altitude. One possible heuristic 
to reduce this type of complexity would be to simply modify certain aircraft (forexample all 

1 ^ S ° f at ^ avoid secto " * which a number oKaft are 

S th c h' 1S u ery Sunilar u° the method curre ntly used. in today’s system. The main problem 

'ZZ^'Sr*? h K OWe r-. 1S u has the P otential t0 be somewhat inefficient due to thematic 
nature of the sector boundaries. An alternative heuristic could be to separate climbing £id 
descending aircraft mto separate streams. 8 a 

7.1.4 Climbing or Descending Aircraft (CoD ) 

3“ c 2™P onent J* a <rOunt of the number of aircraft that are climbing or descending at an instant in 
S-J! 16 s ^ e beunshcs identified for Crossing Altitude Profiles, described abo?e, could bT 
apphed to reduce the complexity due to this factor. Note that there is evidence from the Focus 

chmbM^SmTh sunulations the complexity increases as the number of aircraft in 

l sl j uatlons greases, up to a certain threshold. Beyond this threshold, the comolexitv 
TZ**" t °f e ?f e ^f e i For exam P le ’ ^ al1 aircraft in the sector are descending that traffic ^ 

, ££££“ complex ,han a lraffic si,uadon in which some ™ 

7.1.5 Closest Points of Approach (CPA) 

This complexity measure is a weighting of the number of aircraft that are within a threshold 

rnm™°H° feaCh other at instant in time. Note that the complexity reduction heuristics for this 
complexity component are designed to strategically reduce the significance of this comolexitv 

tn °i! n jf 16 , tar 2 et sector • Thus, a complexity reduction heuristic in this case is not intended 

as ** W ° M be * tacdcal maneuver <“ <«« 

The use of a complexity reduction heuristic such as strategic conflict avoidance is not intended to 

- !h Pr „° Vid ' “ m0m from ESfiS. 

m me context ot the DIRECT System, this heuristic would be weighted bv the imnact of the 

component itself, and compared to toe suggested flight pato 

IS" 5 ° f thC °? er c T plexity reduCtion heuristics ^ incremental movement away from 

51322 Za Z 0V1 i e reuiforce J? ent for the flight path modification suggestions of othlr 

^ebon heuristics. The impact of this complexity reduction heuristic needs to be 

combined wth other heuristics to be effective because flight pato modification is onlv annli^ tn 

move aircraft from one sector into another. In general toe incremental fliffht nath mLr^ l ° 

away from a conflict will not cause toe aircraftto move toto S2 Sr ? modlficatlon 

earliar 

7.1 .6 Aircraft Density (DNS) 
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JS?’ S mgh t P8th SO ^ il avoids sector of hi gh complexity - also applies to the Aircraft 
■ ^ complexity component. However, another heuristic that could be employed would be to 

mcrease the sector airspace allocated to the target sector. This would provide 
theCon trailer to mcrease the degrees of freedom available for aircraft maneuvering. Note P 

hMirichV at t r 1S ^ eiU u StlC does n0t necessaril y guarantee a reduction in the complexity If this 
heunstic is apphed without any other traffic modifications, it is possible for the result to be the 

inclusion of additional aircraft m the sector-a result that could even increase the contribution of thfe 
complexity component to the overall complexity. increase me contribution of this 

7.1.7 Intent Knowledge (INT) 

The level of information about the intent of the aircraft is rated in this complexity factor The 
measurement of this factor is somewhat simplistic at the current time being classified in tn thr^ 

section 4.4.2. The complexity for each aircraft associated with intent knowledge of current 

f ree m «ht is one, and half free flight is one-half. The half free flight case 

to SntS!w^? :iU i ate Pl ? mt ? nt ^formation. This would most likely result in changes being made 
trailer displays and to the communication systems being used for Air Traffic Control . 8 

7.1.8 Aircraft Neighboring Conflict (NBR) 

For each instant in which two or more aircraft are predicted to be within a thrpchoiH 

c«mus made of other aircraft that are width, the gLX. Tdt^^ST'aTa * 

T? COnfhct av0ldance could be employed to reduce the complexity associated with 
« i .. r ', heunstic suggests flight path modifications for aircraft that are considered to be 

a pred l cted < l 0nflict - These ^ P ath modifications will move flSSSSSSr 
aircraft farther away from the conflict situation. neignoonng 

7.1 .9 Conflict Near Sector Boundary (PRX-C) 

This complexity component is a count of the predicted conflicts that will occur within a threchniH 

distance of a sector boundary. One quick reduction of this complodS^^ 

dK ccnfhct locatton so that it was not as dose to the sector bouS^faSta. aSX ' r 

this would require altering aircraft flight paths, thereby reducing the amount of free flIX’nn ’ 
aircraft is allowed Alternatively the nrep/n- amount or rree flight an 

7.1.10 Altitude Variation (VAA) 

This component is a measure of the variability of altitude of all of the aircraft 

compiexi^mfand no 
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7.1.11 H eading Variation (VDF) 

This component is a measure of the variability of heading of all of the aircraft in th<» * 

mstant tn toe. One possible strategy lo reduce the cmplexhy of EffiJbSd JSJSft 

7.1.12 Speed Mix (VAS) 

This component is a measure of the variability of speed of all of the aircraft ; n . 

7.1.13 Aircraft Proximity to Sector Boundary (PRX) 

^SSaSSS Sr 

&&zass£^SSS2&gss 

7.1.14 Airspace Structure ( STR ) 

3?l C r pI S ty com P° n A ent measures the conformance of the traffic flow through a sector to th* 

:S 


7.2 


Operational Procedures and DIRECT Implementation 

a T* il t 


ssawasss :sSS=j i «'f !: 

Resec torization and Free Flight mature to be made once the concepts of Dynamic 

^s^SSSSSSs^WK- 

of to traffic situation in aH ofto secirs that m fEd * m the com P lexit >' 

“^fa,rm^ScTver4“ 
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aircraft to cause the aircraft to enter a different sector than it otherwise would without the flight oath 
modification. The operational philosophy of the DIRECT concept is that the situation witfuna P 
sector is the responsibility of a single Controller. Automated advisories and flight path chanees 
will not be generated automatically by the DIRECT System in order to resolve fsitlLtion within a 

ahlf 1 ^ SCCt0r H Rather ’ g0al ° f SyStem is t0 form sectors within which the Controllers will be 

ab e to provide separation assurance, and can re-assign aircraft to different sectors if necessary to 
allow a reasonable complexity level. accessary to 

The intent of the DIRECT concept is to maximize the free flight flexibility provided to aircraft 
while still maintaining an acceptable level of complexity for the Air TrafficController who must 
assure aircraft separation. Thus, the operational procedures that govern this concept focus on the 
means by which the limitations to that flexibility are communicated to both Pilot, Controller and the 
A-line Operations Center (AOC). These limitations on Free Flight flexibility will £ 

X7^e“SdMbe™. 1 S5 ' St ' m OPer “ 0r ‘ hr °“ gh " eW tafonMtio " “ d Procedural elements, 

7.2.1 Controller Elements 

7. 2. 1.1 Changes to Sector Boundaries 

The DIRECT concept will cause the Controller’s airspace to change to accommodate various traffir 

Cnn^nMprc tlCS 'ii T ? b ® ab l e tG *? andle these airspace changes, the information displays available to 
Controllers will also need to change. For example, sector boundaries will most likelv have tn he 

better ahle'frf? ControUer ’ s In making this information available, Controllers will be 

better able to compare aircraft positions to the dynamic boundary position. Also in addition to 

displaying the current position of the Controller’s sector boundaries, it will be necessary to disDlav 
some detailed information about the future progression of sector boundaries The Controller can ^ 
use this information to predict when an aircraft will be entering and leaving the sector and to 
correlate the future sector boundaries with aircraft flight paths. 

7. 2 . 1.2 Frequency of Boundary Changes 

The frequency with which sector boundaries can be changed is also an important issue to add™, 
e will need to establish a maximum number of sector boundary changes that can be handled in a 
given day-a number most likely constrained by the amount of procedural work that needs to be 
completed in order to successfully alter sector boundaries. In addition; fe nSSS 

^^Tri&nSf 'w^ 0 ** influe " C i ed by the limitations of the users (espidally 

the Air Traffic Controllers) of the system. Pilots may not be as greatly impacted bv contirmallv y 

structures as Controllers primarily because for the mostpart. Pilots will flvfn 
and out of that airspace only a limited number of times throughout a given day Controllers ^ 

wTSlT if SXf ' ' e for «* ‘“rent airspace sweture for entire 8 ho^r 

Z thJr^ i If ^ Str y cture coplmuaUy changes, the Controllers will most likely spend the bulk 
of their tune learning the new airspace structures, rather than controlling traffic. 

7.2. 1 .3 Predicted Aircraft Flight Paths 

“"“P 1 m “ k « Ae prediction of aircraft flight paths and intentions much more 
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7.2. 1.4 Limitations Placed on Aircraft Flight Paths 

TTie DmECT concept will provide aircraft flight path information for select aircraft, designated by 
the Controller. If flight path restrictions are placed on an aircraft by the DIRECT System those * 

,° nS , WlU need t0 be tiWbyed to the Controller. These flight path restrictions win 
generally be high-level, strategic restrictions, designed to keep an aircraft inside one sector, and 
outside of another sector or other airspace area. However, if the controUer does not have this 

^ at ‘°, n ’ S { he ^5 not hav , e ? com P lete understanding of the traffic, and therefore his/her 
workload level would most likely increase. 

7.2. 1.5 Flight Rules (Free Flight or Clearance Based) 

In the event that the ControUer has a mix of free flight and clearance-based aircraft occupying 
his/her sector (as may be the case until aU aircraft are free flight equipped), it win be important for 
the ControUer to be able to quickly make this distinction. Based on our simulations, Controllers 
handle free flight aircraft differently than clearance-based aircraft, primarily by maintaining a 
greater amount of separation space “around” free flight aircraft. If the ControUer is currently 

STJ 1 * mg a * P Tk° d ° f ^ trafflC densit y- k will be important for him/her to know which aircraft 
™ s o tbe b y/ ^ ea te ra m° unt than usuai, to maintain safe se In addition wkh 

1 kfff t0r b^nciaries Controllers wi U need to know which aircraft have the potential (due to 
proaSas possS SeCt ° rS S ° that coordination between ControUers can remain as 

7.2.2 Pilot Elements 
7.2.2. 1 Communications 

Although under the DIRECT System, ControUers wUl most Ukely stiU issue communications 

C f hang fu t0 Pl0tS ’ 11 ma 7 be he lP f ul for Pilots to know that sector boundaries have changed 
the , co " lm ^ lca . tlon frequencies associated with those new sector boundaries may S 
so have changed. If a Pilot is very famihar with a certain airspace configuration (i e shuttle 

San Francisco and Los Angeles), then it is highly likely that the PUot is also 
K W i thC fre 4 uencies associated with each sector. Under the DIRECT System, the dynamic 

T iSfk CSP M ndUlg frequencies wiU need to be communicated to the Pilot to 
ensure that s/he wUl stiU be able to communicate with each Controller. 

1.2.22 Routing Changes 

In much the same way that a shuttle PUot may become familiar with frequently used 
communication channels, it is even more Ukely that the PUot would be famUiar with standardized 
routings between destinations. However, the DIRECT System may suggest that the rou^fSSL 
by aU shuttle aircraft be modified to go around a particu \l sector bound^y, ££d on SS taten 

u nStlCS ‘ ^ “^P 1 ^ toe VORs and fixes which may have been so famUiar to the Pilot 

e no longer used, if only temporarily. In this case, the PUot will need to be made aware of the 
new waypoints used in the modified routings, to ensure a trouble-free flight. 

1.2.23 FUght Rules (Free Ftight or Clearance Based) 

m&iif f Ven ! *5 tl ? ere L is . a of free flight and clearance-based aircraft occupying a 

^ mgh /^ P f ^ If 
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paths to match a sector boundary change, and this information must be made available to other 
pilots. 

7.2.3 AOC Elements 

7.2.3. 1 Proposed Aircraft Flight Path Limitations (For Potential Negotiation) 

At times, the DIRECT System will identify flight path modifications to be made to specific aircraft 
to maintain acceptable levels of complexity across sectors. For example, the DIRECT Svstem 
might su^estthat 2 of 3 aircraft from Airline XYZ (Aircraft #1 and #2) be re-routed into another 
sector. DIRECT could send this information to the Dispatcher for that airline so that s/he could 
compare the suggested modifications against the company’s current goals. It may be the case that 
the dispatcher would rather that Aircraft #2 be remain on course (to meet connecting flight times 
etc.) and that Aircraft #3 be rerouted. This information could be fed back into the DIRECT svstem 
so that the appropriate changes could be made. * 

7.2.3.2 Predicted Areas of Flight Path Limitations Due to Complexity 

Because it will have predictive capabilities, the DIRECT System could also contact the AOCs to let 
tnem know that at some point in the future, certain sectors may be overloaded. The AOCs could 
to P ersonaU y reroute their aircraft around these sectors, thereby reducing the 
chance that the affected sectors would be overloaded, reducing the possible flight path changes that 
their aircraft would have to execute, and possibly reducing their flight times and delay times. 

7.3 DIRECT Concept Exploration 

hi analyzing complexity, we have identified a number of heuristics that could be employed to 
reduce the complexity associated with each complexity factor. These heuristics are- Sector 
Avoidance, Change Conflict Geometries, Create Aircraft Streams (based on speeds, headings or 

T”* 0, M° ve Conflicts, Temporarily Move Sector Boundaries, Increase 
Airspace, and Change Sector Shape. 

A number of these heuristics, however, require that Controllers restrict or alter one or more aircraft 
flight paths in order to reduce the complexity. Although many current Air Traffic Control 
procures result : in aircraft having to deviate or alter their flight paths, these types of strategies will 

w ffifhT ft* environment However, we have seen through our simulations of 

h kf/ 1 * ft at C° ntr ollers will not simply let all aircraft fly under free flight procedures if they (the 
Controllers) are to remain ultimately responsible for ensuring separation. In our simulations ^the 
SjSKf?! e ? ded / Up ^ S1 8nuig flight path restrictions to any aircraft that could possibly result in a 
conflict situation (see Section 6, below). Instead of being able to handle more aircraft, we saw that 

Me to^nlto?” 011111 ° f 00011111,1110041011 somewhat limited the number of aircraft Controllers were 

°ne of the commonly held misconceptions about the DIRECT concept is that it is merely a tool 
which will dynamically alter sector boundaries. However, the intent of the DIRECT project is to 
provide Air Traffic Specialists with both a “Dynamic Resectorization” and a “Coordination 

infT' 0 ' ,!°°' ? no 0011 una S ine man y oases m which only one or two aircraft are problematic 
m achieving the goal of separation. In this case, it doesn’t necessarily make sense to comoleX 

DTRFrT I ^5* a “^° r S1 !f °[’ ^ sector accommodate one or two aircraft. Therefore P the * 
DIRl^ Syston can provide the Air Traffic Specialists with the ability to coordinate the 
modification of aircraft flight paths (for example, the aircraft could be automatically handed off to 
an adjoining sector after the appropriate system-assisted coordination had been established) In 
providing computer assistance for hand-offs, the affected Controllers spend less time coordinatine 
and more time controlling or monitoring traffic. In addition, the DIRECT System couldalsobe 8 ’ 
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used to coordinate aircraft routing with AOCs, to reduce Controller workload air traffic 
complexity, and flight delays. 

The example above describes what the DIRECT System might do if the cause of the complexity 
was only a few aircraft. However, in the case where the complexity of the situation is due to the 
cu ^ e " t fl ^ ht characteristics of a number of aircraft, the DIRECT System might in fact suggest 
modifications to be made to the sector boundaries. For example, if, in the current traffic situation 

* °f 7 craft r “ ag T l 1116 of 1116 the Control e^eSe 

a high level of complexity, then perhaps that complexity could be reduced if the Controller’s sector 
was dynamically redesigned to match the current traffic flow. S Sector 

Of course, sector redesigns would most likely not occur on a continual basis, but it mav be 
ptwsible to identify future traffic periods (arrival rushes or departure pushes, for example) for 
which alternate sector design patterns would be more appropriate. In this case, the DIRECT 
bystem would provide the necessary information and coordination assistance to allow the Air 
Ttafi^pecialisls to effectively and quickly change the sector responsibilites in order to meet the 

h T ge !fi rafflc ' 11 18 to note that not all boundary modifications would occur 

in isolation of modifications made to aircraft flight paths. Rather, the DIRECT System might 
provide suggestions which would result in a slight modification made to the sector boundaries 
combined with some modifications made to select aircraft flight paths. In performing bothif these 
actions, an optimal solution can be reached which reduces the complexity placed on die Controller 

^ mrSPaCe ’ and aU ° WS m ° re ™ crsSt to ^ntinue flyi^^fri ffight ’ 

7.3.1 DIRECT Example 

One of the important goals of our study was to evaluate our complexity measure in sectors other 

thC h < T 5 the8imulatlon - In certain cases, the scenarios used for the simulations 

wa-e substantially different from real traffic scenarios simply because we were to trying to analyze 
specific aspects of ATC complexity. As part of the DIRECT concept exploration.TdeddS to 
ftirther evaluate our complexity measure m other sectors, using uncontrolled aircraft flight tracks 
flying under free flight (i.e., direct flight routes) procedures. g ’ 

As an example of how the DIRECT system and the complexity measure might be used consider 
the followmg example taken from a 6 hour SAR data file (June 6, 1995) from Denve7ARTCC 
S?? “ rc ™g fl y. m g flight (direct) trajectories. For this example, we will be diseasing ’ 

ofS^fo? ?5 d 9 h W t 1Ch are lo f! ted *° * e east of DIA ( see highlights in the figure below) instead 
of Sector 29, which was used for the simulations. Sector 9 is located north of Sector 8. 
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Figure 15 Sectors 8 and 9 in Denver ARTCC 


The following figure shows a detailed view of ZDV8 and ZDV9 for June 6 1995 at 17 39 55 

UTC. Note that the displayed aircraft have been modified to be flying direct flight paths from then 
departure to destination airports. F 1 eu 
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Figure 16 Original Sectorization for ZDV8 and ZDV9 


The two figures below show identical individual complexity factors, as well as the overall 
complexity computation, for both Sectors 8 and 9. Note that in these two figures, the current time 
(as shown in the figure above) is depicted by the left axis. At the current time for ZDV8 the 
overaU complexity is approximately 15.8 (Figure 17). For ZDV9 at the same time the overall 
complexity is approximately the same value (Figure 18). 


omplcviiy 
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Figure 17. Original Complexity for ZDV8 



Figure 18. Original Complexity for ZDV9 


In the above figures, note that approximately 9 minutes into the future, the overall complexity for 
ZDV8 is expected to mcrease to 20.42, and slightly decrease to 10.77 for ZDV9. In addition to the 
expected change in overall complexity, note that the complexity associated with conflict boundary 
proximity (PRX-C), in ZDV8, will be approximately 12.5 (an increase from approximately 10 1 at 
the current time). J 

The figure below (Figure 19) shows the corresponding future traffic situation, using this original 
sector configuration. 6 
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Figure 19 Traffic Situation Approx. 9 Min. Into Future 


Note that there is a large amount of both arrival and departure traffic in the west end of ZDV8 with 
very little airspace available for maneuvering. Also note that there are a number of aircraft which 

are predicted to be flying close to the northern boundary of ZDV8, for the entire time they are in the 
sector. 

The DIRECT system would use the predicted complexity information to suggest alternate (or 
simply modified) sector boundaries that would better suit expected traffic patterns In the current 
example there are a large number of aircraft aiTiving into DIA through ZDV8. However there are 
en ° u | h departing aircraft in the sector to cause some problems. If the system were to be used to 
modify the northern boundary of ZDV8, for example, the Controller would have more airspace 
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available for routing the departing aircraft around the arrivals, thereby ensuring that the arriving 
aircraft would not get any more delay than necessary for in-trail separation. In this example the 
departing aircraft could perhaps be moved a bit northward, without requiring the ZDV8 Controller 
to coordinate the route change with the Controller from ZDV9. 


The figure below (Figure 20) shows this modified sector configuration. Although we haven’t 
shown it in this example, note that the ZDV8 Controller would now have more room near the 
northern border of the sector for routing the departing aircraft. 



Figure 20. Future Traffic Situation With Modified Sector Boundaries 
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P'°f be !° w J sh ^ v resulting complexity measures if the sector boundaries had been 
anged las described. Tfie following two figures have the same time scale as figures 17 and 18 

proximity measure (PRX-C) decreases as well. 



Figure 21. ZDV8 Complexity With Modified Sector Boundaries 



Figure 22. ZDV9 Complexify With Modified Sector Boundaries 
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8.0 


ADDITIONAL CONSIDERATIONS 


8.1 Simulation Debriefing Comments - Future ATC Systems 

The comments presented below, collected after the simulation sessions, provide some examples of 
the concerns Controllers have regarding the transition to a free flight environment. These 
statements are not intended to discourage current efforts towards a free flight system. Indeed 
almost every Controller stated that a free flight system would be possible and workable - as long as 
they were provided with useful and usable tools to help diem maintain separation and situational 
awareness. As new research efforts are proposed and existing efforts continued, it is important 
that we remember to include the system users, both Pilots and Controllers, in our design 
suggestions and decisions. In doing so, we increase our potential to create a system that can 
reduce the complexity placed on the users, reduce the amount of restrictions placed on aircraft, and 
mcrease the safety and efficiency of our nation’s air traffic system. The statements below can ’ 
therefore be used as design considerations to help focus new technologies (such as Controller 
display aids) and aid in the overall modernization of the current system. 


8.1.1 Intent Information 

A number of comments were collected which reflect the importance Controllers place on having 
intent information (from both Pilot and Controller) for effective control. After a particularly 
difficult free flight scenario, one Controller stated: 

• “You have to have some idea of what the Pilot.. .what he wants to do.. .that has got to get to 
the Controllers, that has got to get to the person who is responsible. I’ll tell you what if two 
Pilots were flying along and they were using their TCAS - let’s say TCAS goes out to’ 100 
miles - and they can see that they’re both less than 2000 feet, and that they’re going to get the 
same punishment that 1 would get if they get within 5 miles and less than 2000 feet, the first 
thing those two Pilots are going to want to know is, ‘what’s the other guy going to do?’... You 
have to have some way to have a plan. And when nobody knows the intentions of the players 
involved, then you can’t make that plan...and that’s what we’ve got to get around in free 
flight... we’ve got to get some way to get the intentions of the aircraft.” 

TTiis reliance on intent information was seen throughout the simulations, and it needs to be stressed 
that this intent information must be available in a timely manner. As we saw in our simulations, 
without the ability to look far enough ahead in time to understand the intent of our simulated Pilots 
some Controllers were inclined to individually assign headings and altitudes to all aircraft just so 
that they were able to predict the future locations of these aircraft. This unpredictability of a free 
flight system was a major issue with all of our simulation participants, and they all stated that in 
order to effectively control aircraft in such a system, certain unfavorable precautions might need to 


• Cl: You can only be calculating... or worrying about 2 or 3 of them (potential conflicts) at a 
tune^ You can’t be worrying about 12 or 14 of them. ...And then after you fix it (a potential 
conflict), then you believe it’s fixed and you don’t worry about it.. .you say, ‘this one’s good ’ 
and then you go worry about the other ones that are coming up. But if you have to go back ’ 
each time and worry about whether it’s still good or not, then you’re going to end up with us 
talking to every airplane and telling them exactly what we want them to do through our sector 

unless they are a county away from everybody else.” 

C2: “And since we won’t have time to do that, we’re going to effectively double the required 
separation... I m not going to sit here and tell every airplane what to do twice ...' I want you to 
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nC ,f 30 minutes md then wh en you’re done, you can go back so the next 

Sn«^t U Tt, Can 5 y °u d ° the same ±ing for the next 30 minutes.’ I’m just going to over 
separate them and. ..that s probably not what they’d like.” J g g e 

The level of importance that Controllers placed on timely aircraft intent information shnnlH tv* 
reflated m the design of a free flight system. Whether L changesle hSZra^tto ^ 

STAR^Wh Pment i e '’ F P AD |’ VSCS - etc ) or designed into future systems (such as DSR and 
STARS), the communication of intent is a requirement of Controllers that £ hfo^ J 
There must exist a method for Controllers to be .blew ^teh^ 

order to mamtarn system safety and to be able to sustain effective Control ofTS^,,' system “ 
8.1 .2 Communications and Workload 

Controllers expressed concern about how both their mental and physical workload level, 

equally accountable for violations of that separation. Y * Pdots to be 

Controllers also feel that verbal communications would most likely increase under free flight: 

* 

• We re increasing the amount of actions and decisions we’re having to make anH 

misses some of these things (potential conflicts) whenever you sit down rnd vou larf ^^ 7 


8.1 .3 Conflicts and Route Scheduling 

wwi? — .““I ?" 1 ATC system “ otentimes seen as being overly restrictive and hinhlv 
For example :' tS i[ * ■—» of L strn^e CwS^avoided. 

* Ssfis*. 1“ *& ,eniency 

,Tt h rata^tS^f 1 
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his speed... if he just all of a sudden decides he’s going to speed up because he’s in 

? an m ?!; a f, yOUr com P Iexit y a ^..you could have a 30 or 40 knot overtake all of 
a sudden that you didn’t have before.” overtax an ot 

C2: And there are rules to protect us from that, right now.” 

^, r , 0 " e , s ™ ulation scen ano. we asked a Controller if more detailed flight plan/intent information 

aircrrfuo c^^T i SOrt ° f free mght mles * instead of simply allowing 

aircraft to change heading, speed, or altitude at will. His response indicated that such detail^? 
information might create additional problems: ^ muicatea mat such detailed 

‘‘Strategically, things can get pretty bizarre because of the way the jet stream flows it’ll do 
some rea^y ^d things. What’s their profile look like from the ground? Tma^av VeVe 
g mg to do 28 (FL280) until Denver, then we’re going to climb to 37 until Omaha and then 
we re going to drop back down to 26 because of the winds again until we get east of Chicago 
ttien we’re gomg to climb back to 39.’ That’s an ugly strategic profile from Ae^o^ ’ 

ceruVt^ 011 ? 5 ! 0 ^ I™ 8 - that , t0 . me 811(1 m ing a flight plan that way. But tactically if they 
get up there and start changing their minds like that...I don’t know.” y ’ m6y 

U iS in !P ortant that we P^ent both the situation in which every aircraft is a possible 
H c Jj ^ ^ tent 18 unlai °wn and the situation wherein the Controller is overloaded with 

£ « ^ n ,E lan for every single aircraft. A possible soContoteTiblfmmSe 

routes 

or^„srr™tf^Trritu e not overioadin8 c 0 " «•**» 

8.1.4 Training 

pi! 0 move towards a free flight system, there will have to be a change in the wav Pilot, and 
ers are trained. Current, procedural-based training may no longer be sufficient 

ud “ nM - id « 

fatSeT^H^ SeParaa0 \ S , ,anda^dS “ now ootnpleleljf different You’re t^kg Z 

“"Urey Ye going to take away as nttuth of thftSd^L y 

possible)...but that’s also what we’re trained for.” 06 

If^ la ^ Ship that currendy exists between Controllers and PUots is going to chance with th * 
assume more responsibility for senarpHnn tu V y w i ? uia nKe to see the airlines 
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BACKGROUND DATA 


9.1 The Complexity of Air Traffic Control 

A Controller’s primary task is to maintain separation. To do so, s/he must use aircraft information 
information on the airspace, and any other available resources to effectively control and predict 
potential conflicts that jeopardize this separation. These conflicts can include conflicts between 
two aircraft, conflicts between aircraft paths and airspace, and conflicts between the demand and 
capacity of a particular airport. Air traffic control, with respect to conflict resolution, typically has 
four main processes: planning, implementation, monitoring, and evaluation. These processes, 
along with a discussion of how they are impacted by air traffic complexity, are presented below. 

In the planning process, the Controller's goal is to determine the best course of action needed to 
resolve each traffic conflict. This process typically results in a set of re-routes, vectors, speed 
assignments, altitude changes, coordination with other Air Traffic Specialists, or other control 
actions. However, as part of this planning process, the Controller must also evaluate the impact 
that a given control action, which is intended to solve one particular conflict, might have on the rest 

of the system. Once the Controller completes the planning process and has determined the 

necessary control actions to be taken, the Controller implements the plan through the use of various 
communication and data entry tasks. Although this implementation may be viewed as only being a 
physical task, if the implementation itself requires some sort of planned coordination, then the 
distinction of whether the implementation is a physical task or a mental task is not entirely clear. 
After implementation, the Controller must then monitor the situation to ensure the conformance of 
the situation to the plan, and to evaluate the effectiveness of the plan in resolving the conflicts. 

The complexity associated with these processes stems from the fact that all of the above tasks, 
except, perhaps, the actual implementation of the plan, rely heavily on the cognitive abilities of the 
Controller. Further, each of these tasks is continuously being performed for different aircraft at 
different times, and each of the processes may result in the initiation of another process, as shown 


Conformance with 
or Deviation from 
Goal - Conflict 
Identification 



Figure 23. Mental and Physical Processes Required in Air Traffic Control 
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The “Implement” process is comprised of the physical actions required to carry out a specific nl an 
According to this diagram, this process is indicated by a solid oval and is the only extemallv P 
observable process in air traffic control. The other processes, indicated with dashed ovals are 
internal processes that combine to determine the level of mental effort required for air traffic 
control. According to the diagram above, then, the complexity of ATC is realized through the 

evaluation of the combination of the physical and mental tasks or processes that a Controller needs 
tv perform. 


9.2 Current Vs. Future Complexities 

The complexity tof air traffic control is of particular importance in the study of the free flight 
concept for ATC. The removal of many of the procedures that are currently used for the control of 

fhnSrS, 0 “ a f V °S ated by , the . fr f e u flight wm most hkely affect all of the task elements 

(both physical and mental tasks) that must be conducted by the ATS. For example the process of 

evaluating a traffic situation and determining the conflicts that will arise will most likely become 
more difficult for a number of different reasons. The loss of the current existing organization of 
traffic flows that is created through the use of non-ffee flight ATC procedures will potentially 
increase the number of possible conflicts that might occur. By assigning each aircraft to a specific 
route selected from a finite and relatively small set of routes. t^yfcontroUerTsi^i^ 
reducing the number of locations at which aircraft may come into conflict. Additionally when two 
aircraft are assigned to the same route, they are separated by altitude or by time along the route 
This separation can then be easily maintained and monitored through the use of various methods 

Speed K C ? ntr °f The Controller simply ensures that the distance between the aircraft does ’ 
not decrease below that which is acceptable, by assigning speeds if necessary. 

Two or more aircraft on the same route, with speeds matched to ensure separation, combine to 
orm what is referred to as a ‘stream.’ By creating multiple streams of aircraft, the current ATC 

f l0W C ° ntroUer t0 P rim arily focus on the intersection point of two streams, rather 
having to analyze every aircraft against every other aircraft for a potential conflict As 
mentioned above, separation is easily maintained and monitored through speed control within a 
stream. Between streams, the particular aircraft that may conflict are easily identifiable because 

ootenHd gener *“y onl y he a few aircraft, at most, in each stream that have the’ 

potential to be involved m a conflict situation. The establishment of streams allows a simple 

fUrther redUCeS fte COn,pleXity ( “ experienced by 

flio-ht n b r e JS Ued *5?* both ® valuation and planning tasks will become more difficult under free 
fl f h S® 0 * 05 !? ° f P e mcr , eased flexibility that will be afforded aircraft. Under free 

Im,S. n 5 °r Cr WlU n r °, on | 1 er know the r °nte that an aircraft is expected to follow The 
current RTCA definition of free flight allows aircraft the flexibility of selecting their own route 
speed and altitude, with consideration for aircraft to aircraft conflicts, aircraft to airspace conflicts 

SJS* and SafCty (RTCA ’ 1995) - ™ us ’ a Controller rill be requi^ to^oSr toe 

possibie conflicts that may occur in a region around an estimate of the route that the aircraft will 
follow in toe evaluation and planning process. In this case, toe Controller experiences a 
considerable increase in the number of degrees of freedom that need to be managed. 

The level of difficulty of monitoring an air traffic situation will most likely increase for a similar 
reason. Aircraft have toe flexibility to select their own route under free 

change the route toat they will fly at their discretion. Thus, it will be more diffictotto predicate 
actions and intentions of aircraft. Air Traffic Specialists will have to monitor toe flight ptto of each 
aircraft more closely to determine when an aircraft has decided to change course or f peed. 

Finally, toe implementation task will most likely become less difficult under free flight This is 
because free fhght places much of the decision making process in the cockpit onS^LSTuriess 
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m!n^ ntr0ll ? r mUSt A Ction for akcraft or airs P ace separation assurance, or for traffic 
HSSC- ^ stat ^ above ’ u " der free flight, aircraft will select their own route 

T J S P eciall st5 will not provide route instructions, unless they have been reouired tn 
,.^? n ^ or the previously identified reasons. However, implementation may be auitesirnnip 
very difficult, depend mg on the traffic structure and the goals tf the aircraft q ple ° r 

Since humans have limited processing capabilities, and air traffic complexity impacts nil n f tu* 
processes descnbed above, it is very possible that a Controller canreSKTff fjS flu , , 

of comp exity that is manageable. Therefore, it would prove useful to liable to cr^te a 1?! g vel 
of complexity that would allow us to detennine when a Controlled is appro£h£* 
or her processing abilities. This measure could be used in the current ATC envimnmi^f f 5 1S 
ancVor manage when a Controller will reach his or her processing lintite 

nr^t at m nu me C ° Ul u d P° tentiall y be used to help Understand the impS £*££ flSt 

procedures will have on the air traffic Controllers P m night 


9.3 Previous Work 

9.3.1 “ Measures ’ ' of Complexity 

A number of studies have already addressed the issue of the complexity of an ATC sih.ari™ ff nr 
in-depth review, see Mogford, Guttman, Morrow, and Kopardekar 19951 T n wm? ra ? ff 0n i° r an 

f< ^ used on 40 analysis of the amount of physical work required of an ATS (Schmidf 
1976; Soede, Coetener, and Stassen, 1971- Thornhill iqqsi in Vu^X .TT “ A 1 6 ( Schm idt, 

a measure of physical workload as an indication of the’ llveUf LmpTeiitioHhe ^f 8 °ri USe 

SMS 

mSS XX>^rf on is lhe Seo,or Design “ d 

“ “““ • “ ° f ' “• »■— 

Other studies have used a measurement of the amount of time a Controller spends in 

ST? 1 ?" d ? ther ttrffiwelated factora to cLtte a To wodlKiL to 

r ^^fl^f^aPpn- Suggested applications of his work include the dynamic ^ ^ 

cnSS Stoffm8 of Controllers based on physical workload capacity mtwTcase as 
complexity mcreases, he suggests that additional Controllers may be required 

elem T “ ‘ of complexity 

the number m l n f Stud \ es ’ “ical counts such as 

ssssiit 
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ControUcr performance (Buckley, DeBaryshe, Hitchner, & Kohn, 1983 ). Although performance 
ne f!t sanly be directl y associated with complexity, this work did uncover® strong 
interaction between sector geometry and traffic density that could have implications for ant 
examimng the effects of traffic density on perceived comply im P llcatl0ns for any study 

9.3.2 Two Types of Workload 

While many factors contribute to the complexity of an air traffic control situation thr* c 

company on , the ControUer can be examined in terms of both physical aSd mra U1 *' “ S 
stated above. Throughout this paper, “physical workload" has bj„ “ed to Slotae of 
physical activity reqmred by a Controller, resulting from performing S tofZ simnlv Ti 
ComroUer with his or her operating envhonment. to ote wSs ThS 
I ° f ask f “* Jhos? “shs that are measurable external actions of the ControUer P used to 
implement a plan of action that has been previously determined These tvoes of taclA in^i. a *u 
commumcations and data entry tasks that have been discuSbov? ^ ClUde 

“Mental workload” refers to the amount of cognitive activity spent performing such tasks as the 

h^H^ tm ^JH annm ^ n ? momtonng nece ssary for effective air traffic control The current studv 
ba ; described a method for examining the factors that impact the performance of of * 

untete^dkgof diSeVSr 6 Slgm K fiCant COgniti X e activit y)- ^ has shown how a greater 
thece hi,n a r . diese factors may be incorporated into a measurement of complexity Although 
these two definitions treat physical and mental workload separately problem solvings nH ^ ^ 
resolution typically places demands on both the physical and mental capabilities of the Controller. 


9.3.3 An Incomplete Picture 

Although measurements such as the type and length of physical activity can be userl nc on 
indication of the complexity of an air traffic situation, many studies discount the S thaTlhe 

Mowing scenMio" 611 ' 551 V ^ correct Examples to support this argument are presented in the 

to be commumcated to the aircraft, and may require a re-route t^be ent!^Y^ U1 IJ many clearances 
examples of such tasks are vectoring aircraft ot a standard traffir^of^^i mt ° he system ' 0ther 
approach, making entries to hand m aircraft off to SSher sectw “ “ rcraft for an 

rafeg^^^^ 

“V .ha, 

^ p mng, implementation and monitoring tasks to be performed. 
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in order to create the needed slot. The key difference in this situation is that there is a great deal of 
cognitive activity involved in preparing for one short clearance. The task time and effort needed to 
issue a single clearance will not provide a meaningful measure of the amount of cognitive activity 
involved. J 

As mentioned before, we believe that the dependency on measuring physical activity and inferring 
the level of mental activity may not be the most appropriate method to understand air traffic 
complexity. As in the first example above, previous measures might have identified the situation 
as being complex due to the high level of physical activity required. However, it is likely that the 
complexity of the situation, viewed from a cognitive standpoint, would be considered low. 
Therefore, our complexity measure primarily focuses on the factors of the air traffic situation that 
impact a Controller’s mental processes. 


9.3.4 Measuring Mental Workload 

Theoretically speaking, the “concept” of workload is better defined as a construct. That is 
workload itself is not directly observable or measurable, but must be inferred, based on measures 
and observations of other elements (such as mental and physical tasks) (Mogford, et al., 1995; 
Stein, 1985). The selection of these elements will shape our definition and understanding of the 
workload being inferred. 


Measurements such as the number of communications and data entries, as well as numerical counts 
aircraft have been adopted primarily because this physical data is some of the only direct data 
ttiat is readily available. Directly measuring the cognitive load that is being experienced is more 
difficult and, unfortunately, highly intrusive in a real-world, operational setting. However if we 
maintain the position that simple keystrokes for data entry purposes eventually become somewhat 
of an automatic process, then it remains that the mental calculations and planning work required by 
a Controller is the far more difficult aspect of the job. Therefore, a usefiil measure of complexity 
also needs to consider the details of an air traffic situation that affect the cognitive abilities of the 
Controller, and not just the physical workload. 


This paper does not attempt to define an exact model for measuring the cognitive functions of an 
Alb during control. As well, the work described in this paper was not designed to measure the 
amount of mental workload experienced by an ATS during problem solving (i.e., conflict detection 
and resolution) activities. Although an accurate mental workload measure would be very useful 
and work has been done in this area, it is beyond the scope of this paper primarily because of the 
many problems associated with the measurement of mental workload associated with a particular 
task (Muckier and Seven, 1992; Wierwille and Eggemeier, 1993). Also, as stated in Charlton 
(lyyo), there is very little agreement in the scientific community as to which measures should be 
used to best quantify the level of mental workload experienced in a given situation. 

Therefore, the work in this paper presents a framework and an approach for measuring and 
evaluating the perceived complexity of an air traffic situation, with an emphasis on the traffic 
characteristics that impact the cognitive activity of the Controller. Since we are dealing with the 
perceived complexity involved in an air traffic situation, we are required to communicate with as 

^P ecia l ls 1 . ts ** P° sslb . le m order to get a proper sampling of their perceptions, and 
a better understanding of the complexity associated with their jobs. 
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TJie complexity of an aur traffic situation can not be completely captured by only using the number 
of communications and data entries made while controlling traffic The processes and tasks 

“i m thC , contro1 of ¥* hi ^y cognitive tasks and it is not necessarily true that the 

^Plementation (physical actions) of the results of these cognitive tasks provides a good 
correlation with the complexity of the cognitive processes themselves. Therefore to better 8 
understand the complexity of an air traffic situation, we need to consider the cognitive tasks 
required of the Controller- the planning, monitoring, and evaluating tasks. 

The study presented herein was aimed at evaluating the characteristics of an air traffic situation that 
the cognitive abilities of the Controller. This initial study has providSm!uw Sts ?nto 
e Controller s perception of Air Traffic Control complexity. Although we believe we have a 
useable, initial model of this perception of complexity, it is important to further develop the 

Afrspace! a” “weX* m ’ PaC ‘ ° f S “ Ch “ COmmumcation “> d ^dination, Special Use 

S^f r ? Vel ° Pme , ntS u f the c ? m P lexit y measure will also need to address a number of other 
2™®*- F< t eX T p e ’, the simulation environment only examined one sector in Denver ARTCC 
airspace In order to be able to fully validate the measure, we need to evaluate it in a number of 
other sectors and a number of other ARTCC facilities. The relationships between the Sors in the 

StT. re u 1CW - Since we onl y used Controllers from DenvS A^TCC one 

b ] U u Stlfied m sa ymgthat the measure is only useful in that particular facility. Indeed it 
r 7 Krv^ 0t bC sur P risin 8 if Controllers from the facilities in Washington (ZDC) or New York 

c^lST™nc Co^H™ W 0f h0W “rtual facts conrtbu" .o t overall 
ar^c P l?ff£ * ^L Tr , 'Control. Havmg an mcreased number of Controllers, from different 

ATC m ° re C ° mPlete UnderSUndin ® of how " «— » of 

^ 0rm fr gamed from a vaIida tcd measurement of the complexity of a Controller’s task can 

P ^™ r U e Q s P 118 mea surement will prove even more useful if it can be used in a predictive 

? uch / measurement/prediction will allow traffic management decisionsto be made with 
consideration for the impact they will have on individual Controllers and sectors As well this 
measurement could also be useful for understanding the impact that proposed procedural changes 
win have on the Controller and the ATC system. Fhmlly, acomplexfty^S 

also be incorporated m the development of new ATC automation tools, so that the suggestions and 
y 10018 W0<dd req,,ired ‘° consider "*““"8 complex® of the air 

l ? n °u t f iat m the Previous Work section of this paper, much of the discussion 
focused on what we believe to be the limitations of prior studies While fhece limits 

tteUtled to emphasize the need for the expmment c^XStetlSXaTXSSa. 
this current study was designed to build upon these previous findings Part of nnr in™ ** ^ 

fnnJ’ T nh Cb wiu P*** mf ormation from the current study, is to develop a system that takes 
some of the complexity out of the coordination problem, thereby leaving Controllerrwit? 
time (and resources) to focus on the complexity of the traffic ISw ° C 
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APPENDIX A - DATA SHEETS 


CRITICAL DECISION INTERVIEW QUESTIONS 

COMPLEXITY FOCUS GROUP - BRIEFING 

DETAILED FACTOR INTERVIEW FORM 

FACTOR RATINGS AND WEIGHTINGS 

SIMULATION BRIEFING 

COMPLEXITY SIMULATION RATINGS 
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Critical Decision Interview Questions 


1. ARTCC: 

2. Region 

N NE E SE S SW W NW 


3. Sector Number? 

4. High or Low Sector? 

5. Sector Type? Departure Anival 


Mix 


6 . 

7. 


™nT^i aW T th f ^ CCnari ° 0n the ma P P rovi ded, including the shape of the sector a, 
can recall. Include any special use airspace or weather dlls that were present at ’the 


best you 
time. 


Please describe the situation as best you can recall: 



8 . 


w^heT^DU WoSS'r i h a ,‘ Wha ‘ abou ‘ the *"» situation. 

decision? * y mf0mwtI0n “ lhe Previous expenence to help in your current 


9. What were your specific goals at this time? 


10. How did you resolve the situation? 


Please describe the strategy you used in detail. 


11. Did you consider other alternatives? 
those? 


What were they? Why did you choose not to go with 
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12. What specific events happened that may have contributed to this particular situation being so 
complex? ® 


13. What information was absolutely necessary in being able to handle this situation? What type of 
training or experience was necessary or helpful to make this decision? 


14. What one thing made this situation so complex? If this were different, how might you have 
handled the situation? J 


15. What combination of things collectively made the situation complex? 


16. How much did the fact that you had limited time to deal with the situation affect vour 
perception of the overall complexity? 


17. How do you think you would have handled this situation at an earlier/later point in your career? 
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Complexity Focus Group - Briefing 
Overview 

stud y 1S designed to help us identify potential factors (i.e., traffic characteristic 

^H^h PatternS ’ ? tC - th f at affect y° ur P erce Pdon of the complexity of your job. In the current 

\ h H Se factors wlU ^ used 10 develop a model of the level of complexity 

handll I n8 - m “ traffic sltuation - Our goal is to be able to understand tLcomolexitv 

the CUITently avaUabIe technologies/procedures. In the ^ * 

r£ f U ^ 1x5 used in an experiment to determine if it is feasible to predict the 
complexity of an air traffic control situation. predict tfie 

Mental Versus Physical Workload 

Many studies in the past have used the level of physical workload (number anri/nr leno+h <-.f 

entries> m f “ Me. of 

?S!Z. G i ven i his assump,io * we *** StSUSiSSTL SEZSto' 

de4S,K 

Complexity Factor Identification 

‘° meKUre ' WOrldng *“• We hope » 4 

us better understand the impact these factors have on the D^ceiv^^m^ev r 8 h 7°u t0 help 
an evaluation of these traffic scenarios Also we will f°")P Iexit y of control, through 

factors so that we « «*5 SlSS *' 10 undemand how 10 best i— 
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vanous — “* <0 has 

Airspace Structure (STR) 

StSfeSo! 11 *“ eMmine the tapac ' thal sector size “f stnlclure has on the complexity of 
Please describe the shape and size characteristics of a sector that might be a DartirnWlv 

" ^ nlec^ 16 ' " * — — ~ P p“I»0I 

Why? 

woinH T !f ly ? ffCrent l evels makes sense wi* respect to sector shapes'? For example 
WKte)?Mot? S “ Se fOT US ‘° COmpare t* 0 different sect or shaped (such as narrow vs. 

does the presence of crossing traffic affect the complex^ of control frn thft s“ w? 

Special Use Airspace (SUA) 

Is it simply that they are active? 

Does the size of the SUAs affect the complexity? 

How does the combination of activity level/location/size impact complexity'? 

Sgr" == — ss is,= S r - - 


minutes 

Weather Effects On Airspace Structure (WST) 
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(si “ and sha p=) °f 

sector, and how that translates into increased complexity “ ^ the str "cture of a 

fct^plexity (due to the reduced she and changed shape) increased when a weather ceU is 

does a weather celXJre highe^ sector? If so, where in the sector 

complexity? 8 com P leJUty? Where does it not contribute very much to the 

Do larger weather cel Is make traffic control more complex? 

SeTs“ac^m ta ptoX? f ^ ° f Weather “"*■ "■* locati °" *“* •» sector, and 

far 


minutes 


Proximity of Potential Conflicts 
I nis measure is an examination of the 
sector boundaries. 


to Sector Boundary (PRX) 
location(s) of the potential conflicts) 


with respect to current 


Kofwmplc ^ a>ama l0Cati0ns that are closer t0 sector boundaries result in a higher 

For examples complexity to^er^hen^COTiffim ty ^ thatresultt in high complexity? 
high 6-20 miles, and even less high 21-50 miles? mi CS ^° m b° un dary? Not as 

exksss ss tsssz ss^,' f -r (u - h - * *» 

complexity)? p CI me of factor on the expected 

minutes 


Aircraft Density (DNS) 

A measurement of the density of airtxaf, with respect to the usable amount of arspace. 

Would you say that increased aircraft density results in increased complexity? 

rfSSStfttoS/j* • S “ i<teli “ that we use ,0 “S'S" different weights to different levels 


What is considered high density? Some # of aircraft per hour? 

Low density? ^ ° f nUmber of aJrcraft 1S considered very high density? High density? 
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Very High 

High 

Low 


What would be an appropriate time window to examine for this measure (i.e., how far into 
the future would it make sense to predict the impact of this factor on the expected 
complexity)? 


minutes 


Number of Facilities (FAC) 

A simple count of the number of facilities being served by, or contained within, the sector. 

Our assumption is that an increase in the number of facilities corresponds to an increase in 
the complexity of control. Is this a valid assumption to make? 

What is considered the number of facilities that significantly increase complexity^ (i e 
two facilities? Three or more?) 


Number of Aircraft Climbing or Descending (CoD) 

A simple numerical count of the number of aircraft expected to climb or descend in altitude. Our 
assumption is that if many aircraft are climbing or descending within a sector, then this could 
potentially result in a more complex scenario. 

Is this a valid assumption to make? 

Is there some sort of guideline that we can use to assign different weights to different levels 
of the number of aircraft climbing or descending? 

What is considered a high number of ACFT climbing or descending? Perhaps some % of 
the # of aircraft within the sector? 

Very High 

High 

Low 


What would be an appropriate time window to examine for this measure (i.e., how far into 
the future would it make sense to predict the impact of this factor on the expected 
complexity)? 


minutes 


Number of Crossing Altitude Profiles (CAP) 

Tliis measure is an examination of the number of ascending and descending aircraft profile pairs 
that are expected to occupy (in crossing) the same altitude within a specified period of time in the 
~ ur assumption is that if many aircraft pairs are expected to have crossing profiles, then 
this could potentially result m a more complex scenario. 6 v 


Appendix A 


A-7 



Is this a valid assumption to make? 

Can the levels of this factor be determined with a certain percentage or number of aircraft 
pairs that are expected to occupy the same altitude (crossing profiles)? 

If so, what are the breakdowns of these percentages? 

Very High 

High 

Low 


What would be an appropriate time window to examine for this measure (i.e., how far into 
the future would it make sense to predict the impact of this factor on the expected 
complexity)? 


minutes 


Weather Effects On Aircraft Density (WDN) 

Weather also impacts the density of the aircraft in the sector, because the amount of available 

airspace is reduced. Therefore, this measure will examine the impact that a weather cell has on the 
density of aircraft. 


Is complexity due to density increased when a weather cell is present? (i.e., a weather cell 
reduces the amount of available airspace and therefore, the same number of aircraft will be 
considered to be a higher level of density.) 

What would be an appropriate time window to examine for this measure (i.e., how far into 
the future would it make sense to predict the impact of this factor on the expected 
complexity)? 


. minutes 


Variance in Aircraft Speed (VAS) 

A measurement that looks at the variability of speed tracked for each aircraft. 

We are Msuming that if all of the aircraft are going the same speed (e.g., 250 KIAS in 
lower ARTCC sectors), then it might be easier to deal with because you don’t have to 
worry about the mix of speeds that need to be managed. Is this a valid assumption? 

h there a range of speeds that you consider to be generally the same? For example the 

220 “l 230 HAS might not be that great, but the difference between 
180 and 230 HAS might be considerable to affect the complexity of the scenario. 

Is this range the same at high and low speeds? 

What would be an appropriate time window to examine for this measure (i.e , how far into 
ImplSy')?^ U makC SCnSe t0 predict the of this fact or on the expected 
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minutes 


Variance in Directions of Flight (VDF) 

A measurement that looks at the variability of direction for each aircraft to be controlled. 


don^fhave'todeal 

S 5a^t;a.‘&»rr-. 


minutes 


Performance Mix of Traffic (PRF) 

IfoSt"™"" that looks at the variance in perfomtance capabilities of current and expected 


We are assuming that if all of the aircraft have relatively the same performance 


minutes 


Winds (WND) 

5TSSisul thC Wi " d ^ “ ZimUth by *“*“*• *“» iB ™Pact on aircraft performance 


In general, what level of wind speed starts to somewhat impact aircraft performance? 
In general, what level of wind speed starts to Significant impact aircraft performance? 

=H3!F“" sxxfcssx s, . 


Distribution of Closest Points of Approach (CPA) 

couldT^S^ (laterally) fUgh, paths whrch 

oSsIy'^e 0 h" '$£££• 


Appendix A 


A-9 


like two aircraft are going to cross within 8 miles of each other, do you still take some 
action? What is the approximate limit for which you will take action? 

What would be an appropriate time window to examine for this measure (i.e., how far into 
the future would it make sense to predict the impact of this factor on the expected 
complexity)? 


minutes 


Angle of Convergence in Conflict Situation (ANG) 

A measure that examines the predicted angle of convergence in a conflict. Shallower angles of 

convergence result in a longer period of potential conflict, so we are assuming that this might result 
in a higher level of complexity. 6 


Is this a valid assumption? 


Is there a specific angle (or angles) that could be considered a cutoff point(s) for different 
levels of complexity? For example: 1* - 30’ angle of convergence could be very high 
complexity, 31 -60 is high complexity and 61* - 90’ is low complexity 


Very High 

High 

Low 


What would be an appropriate time window to examine for this measure (i.e how far into 
the future would it make sense to predict the impact of this factor on the expected 
complexity)? 


minutes 


Neighbors (NBR) 

The proximity in lat. and vert, distance between aircraft pairs in conflict and other aircraft within 
some parameter distance or tune. 


Is it meaningful to simply say that the presence of neighboring aircraft 
complexity? 


impacts perceived 


How close do these neighbors have to be in order for them to impact the complexity? Are 
there varying levels of distance that could be assigned different weights with respect to how 


Lateral Distance 


Vertical Distance 


Very High 

High 

Low 


Very High 

High 

Low 
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What would be an appropriate time window to examine for this measure (\ e r ■ 

wmp£dty"? Uld “ m,ke 10 predic ' lhe of this f »«°r « «he expected “‘° 


. minutes 


Level of Knowledge of Intent of Aircraft (INT) 

of fte 0f 1 - ■ - -tart I- oc tee complexity 

of “ • tart - ,he 


Separation Requirements (SEP) 


minutes 


Coordination (CRD) 

(wi * , 

a valid 


minutes 
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Factor Ratings and Weightings 

Overview 

importaI^£a?^upLre on^A°^^con^l^ty facto^*^a^^would Hke^or'voirtod 6 ^ 

you feel tha factor contributes to the complexity of an air traffic situation. g y 

• a j° r i r St sectl0n this questionnaire, we ask that you consider each factor 

“ ked ‘FiSTv^f ' “Pf" toSSfSS of M«^cSp°e^°fa«“s beP 

r,^ri£SS5 

change one of your decisions m,ght ** helpful to use 8 P 8 "^, “ case you 

and cha"ge a y„^tcS™^tr./l‘ hiS <1UeSli0 " naire - >»“ “« <» go back 

s^SSSS^ 

Section 1 - Absolute Levels of Complexity 

S^SSs^^fSS^SSSSS. 

=s=:srri-SS3 J ““- 

pairs of faSnThe SSeSatSta S S w° Sl f*™' combinations of 

^nuS a ils1g“\tigo U f f^' TS T“ o" 

die complexity .^Please Sue vL eXd tf K K* 1 d ° " 0 * have mui:h ™P»« on 

srSaaaessS^&g* 
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Single Factor, Absolute Ratings 
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Section i 2 - Absolute Levels of Complexity for Factor Pairs 

For the next section of this questionnaire, we ask that vou rate each fnrtnr noir „ 

for the inthviduS factors. m, is, rate S pa^bS onlow £3? 
you feel those two factors, when experienced at the same time, contribute to the complexity oAn 
mr traffic situation. Again, as an example, you should assign a rating of “10” to anv Dair of farm™ 
combln « i ' y°“ f«i asato hnpact the level of c^plexityyou expeneS Zn 
f ^' n * “ a,r tI | a ® c Slluau ° n Conversely, you should assign a rating of“l" to any pair of 
hlfln* ***** y0U have hthe impact on the complexity of a situation A rating of “5” should 
stoS a F P Zl?„ yOU feel onl> : ^a wh at impact the overall clptexhy of a 

nSsr^sa tsssg&sz 

The acronyms used in the table are the same as before, and you mav refer to them at anv 
rated. ^ m number (0 ' 10 ) in each s P*ce corresponding with the factor pair being 

SIR SUA WST PRX rns rao rwh r * ad r.x^T 


ill 

i;W# 






iiii; 


mm 


\<U : % 

■ Jj» . I; 

liliifil 


WM 


Si 

ili 


W8& 


iliiil 


ill 




MB 


®”Hwns i x[u)«Mw,rii rTOa , wi;MSipaii 
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Section 3 - Relative Levels of Complexity 

contribution of each o?X^^^ Uk " you *> rate the relative 

«*> to to emssi 'ThP 1 '' “If factor tot you 

factors) should be given the rating of “1” The ftch^O^ hS^ 00 ( ?° Ve ^ other Usted 
complexity (above all other remaining factors! Shk!«? *e second greatest impact on the 
difficult task, given the numtooffe^ ratm * of , 2 ” Although this is a 

.o XX. I, might ta helpM t0 use a 4S fr0m 1 


Single Factors, Relative Weightings 
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Simulation Briefing 


Overview 

rt^/ 5 «^ USly describ ^f to y°“* ** focus of study is to be able to measure the different factors 
that affect your perception of the complexity of your job. We have developed a model of the level 
of complexity associated with handling an air traffic situation, based on the information you gave 
us when you last visited Wyndemere. The simulation sessions being held today will be usecfto 
validate our model. For these sessions, we ask that you participate fi a numberof simulation 
scenarios and to evaluate the complexity of those scenarios. We will ask you to rate both die 
overali complexity of the scenario, and the complexity of the scenario based on individual 
complexity factors. In addition, we will ask you to describe the specific complexities of the 
scenario, and ask for your comments on our computed complexity values. 

^n h »rii y< S, Wi11 C ° ntro1 0f seven (6) scenarios - first scenario will be a calibration 
scenario. This scenario will be presented to familiarize you with the simulation environment- 

including the communications system and the design of the sector. You do not need to fill out anv 
questionnaires after these this scenario. 1 any 

After a short break, we will present you with the five (5) test scenarios. For these test scenarios 
iroffir* y * U ajs s ^ me the role of a Radar Controller, and one of you will simply monitor the 
l”®® sltu » tlon (Radar Monitor). The test scenarios will last about 20 minutes each After each 

hS°’ y ° U W u U b °? 56 glven a q uestionn aire to complete, and will be asked to participate in 
a short discussion about the scenario. This discussion will be audiotaped so that we may review 

ST? ater ’ When we are m&l y zin & the data. In total, each test case (simulation scenario 
complexity ratmg questionnaire, and discussion) will last approximately 30 minutes. 

Radar ControUer has completed all five (5) test scenarios, the Radar Controller and the 
Radar Monitor will switch positions-the person who was the Radar Monitor in the morning 

Sr W ? n° W TTT he Radar Controller. The simulations will proceed as above new 

PreSenKd With the “ libraa0n SCMari0 - “ d then P f0Ceed <0 

TTiere are three conditions under which the simulations will operate. For the sake of convention 
^e three conditions are referred to as: Current Procedures, Half Free Flight Procedures and ’ 
Full Free Flight Procedures. A description of each of these conditions is presented below.’ 

Conditions 

( C)yrr<?nt t^Q^ires . In an attempt to simulate current ATC procedures, you wifi be presented 
wiA aircraft flying on preferred routes, and will be given full flight strips L alTS[ For this 
condition, aircraft are required to request ATC clearances for any desired routing changes. 

To simulate the “Half Free Flight” portion of the simulations 
TrhonT m T T" 8 direct rout< * 'tween tiie origination and destination airports We are going 
to change your short-term intent knowledge by allowing aiicraft to vary their heading within^ on 
mde (10 mdes to each side of their “direct” flight plan) ' ^corrid^r,” 

TT TT 011 0f tf ? u ’ aSS1 i gned ^titude, without requiring clearance. The aircraft aTstill reauired 

°" s “ ““ avoid ““- whidl wou,d most SJSSS 

r th depmWt a " d ^val P anpo“df„S ' be 

given the assigned altitude and current speed for each aircraft being controlled. y 

T e FuU Fr u e Fllght condi tion wiU present you with aircraft flying along direct 
ght routes, but in this case, the aircraft are allowed to change heading and/or altitude as dLired, 
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S! h0U i neCeSSarily IT™ 8 clearance - F °r this condition, the only intent information given to 
you is the origin and destination airports, presented on flight strips. 

Complexity Factors 

k aS'staijafon” 18 ” 5 a lisdn8 and a descri P don of the complexity factors that we are measuring 


Aircraft Density (DNS) 
Aircraft Count (ACT) 


Conflict Neighbors (NBR) 


Distribution of Closest Points of Approach (CPA) 


Angle of Convergence in Conflict Situation (ANG) 


Proximity of Conflict Aircraft to Sector Boundary 
(PRX-Q 

Performance Mix of Traffic (PRF) 


Variance in Aircraft Speed (VAS) 


Variance in Directions of Flight (VDF) 


Special Use Airspace (SUA) 


A measure that looks at the effects that the knowledge of 
intent of an aircraft has on the complexity of a conflict 
involving that aircraft. 

A measurement of the density of aircraft with respect to re- 
usable amount of airspace. 


A simple count of the number of aircraft that need to be 
controlled. 

This measure is an examination of the number of ascending 
and descending aircraft profile pairs that are expected to 
occupy (in crossing) the same altitude within a specified 
~~ riod of time in the future. 

A simple numerical count of the number of aircraft 
expected to climb or descend in altitude. 


The proximity in lat. and vert, distance between ACFT 
pairs in conflict and other ACFT within some parameter 
distance or time. 


pis measure is a time-based distribution of the number of 
intersecting (laterally) flight paths which could be potential 
conflicts. 


A measure that examines the predicted angle of convergence 
in a conflict. Shallower angles of convergence result in a 

;er period of potential conflict 

The impact that the location(s) of the aircraft with respect 
to current sector boundaries has on the complexity of 
control. 

An examination of the location(s) of the potential 
conflicts) with respect to current sector boundaries. 

A measurement that lodes at the variance in performance 
capabilities of current and expected aircraft 


A measurement that looks at the variability of speed 
tracked for each aircraft. 


A measurement that looks at the variability of direction for 
each aircraft to be controllfH 


This measure is intended to identify how the 
number/size/activity of restricted areas, warning areas, and 
military airspace impact the complexity of an air traffic 
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Complexity Simulation Ratings 

fact0r from 1 10 10 > based on how strongly you feel that factor 
ontributed to the complexity of the scenario. You should assign a rating of “10” to anv fart* r th 

you feel pafly impacted the level of complexity you expertenaxi and a^adnvcf' l ? / ? 
that you feel had yaylitfc impact on the tLtptodly offeS ‘h„"2 ?° ,0r 

given to a factor which you feel onlv snmmhat imno/'tA/i fUck it should be 


of 


frorn \Z ^ Ptee rate •» (again. 


on a scale 


Structure 


knowledge of 



Intent of 
Aircraft 

Aircraft 

INT 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

o 

Density 
Aircraft Count 

DNS 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

o 

_ Number of 

ACT 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

o 

Crossing 
Altitude 
Profiles 
N umber of 

CAP 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

o 

Aircraft 
Climbing or 
Descending 
Conflict 

CoD 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

0 

Neighbors 
Distribution of 

NBR 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

0 

Closest Points 
of Approach 

Angle of 

CPA 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

0 

Convergence 
in Conflict 
Situation 
Proximity of 

ANG 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

o 

Aircraft to 
Sector 
Boundary 
^rox unity of 

PRX 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

o 

Conflict 
Aircraft to 
Boundary 
fterformancc 

PRX-C 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

0 

Mix of Traffic 

Variance in 

PRF 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

o 

Aircraft Speed 
P Variance in 

VAS 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

o 

| Directions of 
Flight 

Special Use 
Airspace 

VDF 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

0 
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Appendix B - Complexity Focus Group Data 


INTRODUCTION 

LEVEL OF KNOWLEDGE OF INTENT OF AIRCRAFT [INT] 
AIRCRAFT DENSITY [DNS] 

NUMBER OF AIRCRAFT CLIMBING OR DESCENDING [COD] 
DISTRIBUTION OF CLOSEST POINTS OF APPROACH [CPA] 

NUMBER OF CROSSING ALTITUDE PROFILES [CAP] 

PROX. OF POTENTIAL CONFLICTS TO SECTOR BOUNDARY [PRX] 

ANGLE OF CONVERGENCE IN CONFLICT SITUATION [ANG] 

COORDINATION [CRD] 

NEIGHBORS [NBR] 

SEPARATION REQUIREMENTS [SEP] 

VARIANCE IN DIRECTIONS OF FLIGHT [VDF] 

AIRSPACE STRUCTURE [STR] 

PERFORMANCE MIX OF TRAFFIC [PRF] 

SPECIAL USE AIRSPACE [SUA] 

NUMBER OF FACILITIES [FAC] 

VARIANCE IN AIRCRAFT SPEED [VAS] 

WINDS [WND] 


....B-2 
....B-5 
....B-6 
...B-7 
...B-8 
...B-9 
. B-10 
. B-12 
. B-13 
. B-14 
. B-15 
B- 16 
B-17 
B-18 
B-19 
B-20 
B-21 
B-22 
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Introduction 

The data from the absolute factor (and factor pair) ratings, the relative rankings and the 
ocus group interviews is presented below. As part of our study, we asked controllers’ to consider 
the impact that weather has on the complexity of control. However, since we do not have reliable 
weather information to include in our simulations, we are not going to measure the impact of 

US® V\* 1S Cl f ren ^ of research. Therefore, in the data table of sorted absolute ratings, 
presented below (on the left), the weather data has been removed. For the relative rankings of the 
t ? rS ’ P r ® sented below sorted by z scores, the weather data remains in the table due 
, , e fact that if weather was not considered in the original rankings, the relative relationships 

h thCr factors .! na y have been different. The relative importance of weather, as shown 
in the table, however, will not be discussed. 


I Absolute j 

Factor 

It 

of” 

INT 

7.9 

2.18 

DNS 

7.2 

2.39 

CAP 

7.2 

2.04 

NBR 

6.7 

2.11 

CRD 

6.7 

2.45 

CPA 

6.5 

1.78 

CoD 

6.4 

2.07 

SEP 

6.3 

1.70 

PRX 

6.0 

1.94 

ANG 

6.0 

1.89 

STR 

5.2 

2.66 

VDF 

5.1 

2.13 

PRF 

5.1 

2.51 

FAC 

5.0 

2.49 

VAS 

4.3 

2.31 

SUA 

3.9 

2.02 

WND 

3.2 

1.75 


| Relative 

Factor 

z Score 

WDN 

1.14 

WST 

1.01 

INT 

0.64 

DNS 

0.57 

CoD 

0.46 

CPA 

0.41 

CAP 

0.23 

PRX 

0.16 

ANG 

0.12 

CRD 

-0.14 

NBR 

-0.18 

SEP 

-0.21 

VDF 

-0.28 

STR 

-0.37 

PRF 

-0.41 

SUA 

-0.60 

FAC 

-0.68 

VAS 

-0.75 

WND 

-1.12 


In the following pages, each factor will be examined according to its placement in the relative 
ranking scale, excepting the weather information. For each factor, a short summary of the 
qualitative interview data will be given, and information regarding its relative importance to 
cwnplexity as well as an approximation of a weighting will be described. Whenpossible and 

on the “” 0Um ° f time rcq “ ired “ ^ °f that faceor 
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Di addition, to determine how the individual factor weightings should change with the presence of 
an additional, influencing factor, we decided to examine the difference between the absolute ratines 
given for each [individual factor and the absolute ratings given to that factor combined with every 
frAPi i v C t0 rrf ;' te f sts , were P erf ormed on each distribution (i.e., [CAP] vs. [CAP x STR], 

J. ; i s - X U ^ UA J’ etc -)’ assuming equal variances. For statistically significant results 
(i.e., the two distributions were found to be significantly different), the t-test results will be 

onH S fhl ted T* C ! 1SCUS i ed - ^^ddidon, possible reasons for why a significant difference was found 
and the implications that the finding has on the measurement of complexity will be given An 
example describing this process is given below: H y & ■ nn 


Single Factor Ratings 


CAP 


Si S2 S3 S4 

8 8 9 6 


SS S6 S7 S8 S9 S10 
3 9 7 6 10 6 


M; q*- 1 * 

7.2 2.044 


Factor Combination 
Ratings 

CAP x STR 
CAP x SUA 
CAP x PRX 


SI S2 S3 S4 SS S6 S7 S8 S9 SI Q 

8 9 6 5 3 7 7 i 7 

3692549451 
89108 6 3 9 6 5 9 


a*-'* 


5.6 2.3664 

4.8 2.6583 

7.3 2.2136 


4 S1 ? nifl ^ lt Terence between the absolute ratings 
Sn u CAP d h u absolute ratm « s assigned for [CAP x SUA] (t = 2 . 26 ; p < 0 037 two- 

S^LI?rf Ver - we "Z** ft abs ° lute rating of comptodty associated with the combination of 
• ^? e . factors is significantly lower than the absolute rating of complexity assigned to the 
mdividual factor [CAP] alone. A possible reason for this might be explained by the fact that SUA s 
are usually not located in the direct path of a portion of airspace. If this should happen however 

SUA°and al[ dSlS t0 hav % f ° r exam P le ’ aU cubing aircraft to go around the^orth side of the 
XfVhof d U d S ng T Craft g0 81-01111(1 the south sid e of the SUA. Therefore the complexitv 
of that scenano would not be considered as great an impact on complexity as the presence of a ^ 
large number of crossing altitude profiles between two aircraft. c presence or a 

However an mterestmg problem is highlighted through the examination of this example From an 
^ ltl 7 e standpoint, it would make sense to see an increase in absolute complexity when in the 

i mT "ft ° rossmg P rofUes t CAP J and presence of a Spiral Use 

Airepace [SUA] simply because of the fact that mere aircraft route changes would be required 

nart!Sn er, f the faCt ft! there .. w l as a significant decrease in this rating leads us to believe d!at the 
participants may not have all been usmg the same criteria for assigning factor weightings It 
became apparent to us that in comparing the quantitative, numerical data for the combined ratine 
^ n qUahtaUve ^“^ **"■ controllers may have 

based on an assumed (but unknown to us) relationship between those two factors. S 

This fact is further evidenced upon examination of a simple correlation matrix between individual 
factor, absolute ratings. In many cases, the correlations obtained do 

correspondence to the combined factor ratings. Although no statistical tests ~ ! , 

SST?™; “ curs !"y ““““ion of data doShidieate that controller may hav e g 

JJJSKSJ thoexisten ^ of additional relationships between factors when assigning ^combined 
Tbus ’ in identifying weightings for combined factors, we decided to use the more 
J but - dlff i CU t to quantify, interview data as a basis for our assignments Due to the nature 
of the interview data, we realize that a certain level of researcher subject can !mpa£ Se 
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“ S ,'*"^„ we ' ght \"8 s However, since this study is designed to allow us to further modify our 
SSlStf t^eritT ’ We fed that *“■ “ “ acce P“ ble s«P to defining L 
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Level of Knowledge of Intent of Aircraft [INT] 

Absolute Rating: 7.9 

at ** 11131 ^ knOWledge ° f 11116111 ° f 311 aircraft has 0n of a conflict 

During the complexity focus group interviews, the discussions about aircraft intent information were very 

f *' **' did DOt h8Ve tbaA “ tent formation (with respect to chZes in 

^ ^ admgS ’ etc ) ^ ^ "’“Piexity would become very high. In one controller’s opinion control 

Zl^T ' U l mitely h M der ”. 111 genera1 ’ il is telieved controllers may have a difficult time imagining a 

10. 15 mtautes) ataaf, latent Mar™** 

TWore. during the “Free flight” portion of the simulations, we will affect the short-term intent knowledge of 

22 TT* to V 7' their headin 8 within a 20 mile (10 miles to each side of their “direct” flight 

p ) mdor. As well, the aircraft will also be allowed to change their altitude bv 500 feet in either HJww* r 

on ft^cS m '° n ° al,0n - WhiCh U “ “ COrfa ^ Wilh ' ie of ^ PK« 


Cdmbmmg the impact of intent information with the other factors, the average data is presented below The high 
level of complexity associated with intent information combined with the proximity to sector boundaries rPRX^the 
nS^APT fTT ^ [VDF1 - ^ PreS£DCe °f neighboring aircraft &BR], ti nuTfc^oW^ S 

fSEPl the V °T ? a P Pr ° aCh [CAP] ’ ^ P® 1 * 0 ™ 1 ®™* mix ^ baffle [PRF], the separation requirements 

table, and wd, ^ [ANG > * —X ~ » ** 


PRX 

8.2 

1.93 

VDF 

8 

1.83 

NBR 

8 

3.09 

DNS 

7.9 

2.33 

CAP 

7.8 

1.48 

CPA 

7.5 

3.03 

PRF 

7.3 

1.89 

SEP 

7.2 

3.05 

CoD 

7.1 

1.60 

ANG 

7.1 

3.03 

VAS 

6.9 

1.97 

SUA 

6.2 

3.68 

CRD 

6.2 

3.33 

WND 

6 

3.83 

STR 

5.9 

3.35 

FAC 

5.6 

3.57 




Appendix B 


B-5 



Aircraft Density [DNS] 

Absolute Rating: 7.2 

A measurement of the density of aircraft with respect to the usable amount of airspace. 


t fTSi f C °T T*? *5" “ mCrcaSe “ ** de ° sity of “““ft within a specie amount of airspace results 

m “ 1 116 COmplex , 1 i ty (or P° tential complexity) of a situation. The reason the potential complexity!^ 

emphasized is because conuoUers stated that the complexity due todensity also depends onXSe^rS doin, 

without JLr ° r CXamp e ’ ^ ^ aircraft s^Pty fiy^g through the sector in the same general direction 

« ,o 

< ' d “* 


upper limit to what a controller can deal with is somewhere around 30 - 35 airatft 


Some controllers would like to know 
time. 


about a significant increase in aircraft density at least 15-20 minutes ahead of 


PRX 

CoD 

INT 

CAP 

NBR 

CRD 

STR 

SUA 

SEP 

CPA 

ANG 

VDF 

VAS 

PRF 

FAC 

WND 


8 

1.41 

8 

2.36 

7.9 

2.33 

7.6 

1.96 

7.4 

2.80 

7.3 

2.79 

7.1 

2.28 

7.1 

1.60 

6.9 

2.96 

6.8 

3.01 

6.8 

2.39 

6.7 

2.87 

6.3 

2.83 

5.9 

2.42 

5.4 

3.57 

4.1 

3.14 


t-Test Two-Sample Assuming Equal Variances 
DNS vs. DNS x WND 



Variable 1 

Variable 2 

Mean 

7.2 

4.1 

Variance 

5.73333333 

9.87777778 

Observations 

10 

10 

Pooled Variance 

7.80555556 

Hypothesized 
Mean Difference 

0 


df 

18 


t Stat 

2.48110296 


P(T<=t) one-tail 

0.01160034 


t Critical one-tail 

1.73406306 


P(T<=t) two-tail 

0.02320068 


t Critical two-tail 

2.10092367 



Here again, we see a justification for why the statistical data mav not hp a i 

There is no reason to believe that the presence of wind fWNDl rXnhirwt ^ 8 ^ , Way ° aSSlgn we, g htin gs- 

the absolute level of complexity ^ ‘ denffl f y mS] wouJd reduce 

were no additional amount of complexity added bv the associat ^ witil ^sity alone. If there 
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Appendix B 


Number of Aircraft Climbing or Descending [CoD] 

Absolute Rating: 6.4 


A simple numerical count of the number of aircraft expected to climb or descend in altitude. 

H0WeVer - "‘“P between 

of conflicting alitade pairs [CAP), the tains 1 “S* ® NSI ' ““ nUmbe ' 

general, most controllers stated that if vou arc confrnllino «„ • , ’ T* type rf sector worked [STR], In 

increase in the number of aircraft ***** ** ° f “ 

Most of the answers given in the interviews SS £?*“ t T "T worbn S 811 ov <^t sector, 
overflight sector, and some existing condition is doming aimraft nSS C °° m “ a: WK worki "S “ 

*“”■* ^ ** «• be »e^ 

numbers below of coarse denend™ uT^ 6 P 61 ^^ of “* total number of aircraft The 

controllers assumed "moderate” levels of ^ <“ “ DNS) ' ^“e- 




a (ml) 

Very High 

>52% 

14.6 

Hicrh 

>31% 

8.6 

Low 

<23% 

6.5 


easier Incuse 

minwes is reasonable to detennroe Urn number of atari, ahead about 15 


DNS 

CAP 

INT 

STR 

NBR 

PRX 

PRF 

VAS 

ANG 

CRD 

FAC 

CPA 

VDF 

SEP 

SUA 

WND 


8 

2.36 

7.1 

2.13 

7.1 

1.60 

7 

2.11 

6.6 

3.06 

6.3 

2.71 

6.2 

2.62 

6 

3.50 

6 

2.45 

6 

3.16 

5.8 

3.22 

5.8 

2.62 

5.7 

3.59 

5.6 

2.07 

4.9 

3.07 

4.3 

2.21 


t-Test Two-Sample Assuming Equal Variances 
CoD vs. CoD x WND 



Variable 1 

Variable 2 

Mean 

6.4 

4 3 

Variance 

4.26666667 

4 Q 

Observations 

10 

“.7 

10 

Pooled Variance 

4.58333333 

Hypothesized 
Mean Difference 

0 


df 

18 


t Stat 

2.19337847 


P(T<=t) one-tail 

0.02082716 


t Critical one- tail 

1.73406306 


P(T<=t) two-tail 

0.04165432 


t Critical two-tail 

2.10092367 
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Distribution of Closest Points of Approach [CPA] 

Absolute Rating: 6.5 


This measure is a time-based 
potential conflicts. 


distribution of the number of intersecting (laterally) flight paths which could be 






a M> 

Concern 

<13 

I 2.27 

Action 

<8.2 

ri.09 


SdL™ '^ s TiTm~.'Zfl n T Sary '°,f amiac * POKod^ conflict 


£ ^ b trrr • (mTl 

dantaaon of closest points of approach tcsnlts in relatively high^S 


INT 

CAP 

DNS 

PRX 

NBR 

CoD 

VDF 

SIR 

CRD 

SUA 

VAS 

FAC 

SEP 

PRF 

ANG 

WND 


7.5 

3.03 

7 

2.91 

6.8 

3.01 

6.7 

1.95 

6.1 

2.56 

5.8 

2.62 

5.8 

2:30 

5.6 

2.32 

5.6 

2.76 

5.4 

2.55 

5.4 

2.17 

5.3 

3.13 

5 

2.36 

4.9 

2.33 

4.6 

3.37 

4.4 

2.67 
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Number of Crossing Altitude Profiles [CAP] 

Absolute Rating: 7.2 


This measure is an examination of the number of ascending and descending aircraft profile paks that 
occupy (in crossmg) the same altitude within a specified period of time in the future. 


are expected to 


“ S tl f t “ “ crease * d* aumber of aircraft pairs expected to occupy the same altitude (in crossing 
results m a higher level of complexity. During the interviews, the controllers stated that is somewha dependent 

£ S -• —• - *- •SSS 


Looking at a percentage matrix (as above), the foUowing information was 
on the density of the aircraft. 


collected. Again, this percentage is based 




o (m,i 

Very Hiah 

>37% 

13.3 

Hiqh 

>25% 

5.5 

Low 

<19% 

6.6 


fhTrt. Vf tr ? UerS -. S .‘t ted 11131 15 ' 20 minutes “ 8 rcasonable amount of time to look ahead to examine the impact 

^ 011 COmpleXity ' ^ reaSOT numbere - “ of percentages SZSS yZS 
than those of the previous question is because of the simultaneous mental calculations required foTeach nair A, th* 
number of aircraft pair increase, the calculations get more difficult at a faster rate. P * ** 


INT 

7.8 

1.48 

DNS 

7.6 

1.96 

PRX 

7.3 

2.21 

CoD 

7.1 

2.13 

PRF 

7 

1.94 

CPA 

7 

2.91 

NBR 

6.7 

2.87 

SEP 

6.7 

1.77 

ANG 

6.6 

2.59 

VAS 

6.4 

2.91 

CRD 

6.1 

3.07 

VDF 

5.7 

3.43 

STR 

5.6 

2.37 

FAC 

5.3 

3.16 

WND 

5.2 

2.90 

SUA 

4.8 

2.66 


(■Test Two-Sample Assuming Equal Variances 
CAP vs. CAP x SUA 



Variable 1 

Variable 2 

Mean 

7.2 

4.8 

Variance 

4.17777778 

7.06666667 

Observations 

10 

10 

Pooled Variance 

5.62222222 

Hypothesized 
Mean Difference 

0 


df 

18 


t Stat 

2.26330061 


P(T<=t) one-tail 

0.01810682 


t Critical one-tail 

1.73406306 


P(T<=t) two-tail 

0.03621364 


t Critical two-tail 

2.10092367 



The possible reasons for. and the implications of. 
document. 


this finding have been described above in the introduction to this 
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Prox. of Potential Conflicts to Sector Boundary [PRX] 

Absolute : 6.0 


This measure is an examination of the 
boundaries. 


location(s) of the potential conflict(s) with respect 


to current sector 


fa geroral controlters feel that when a conflict is expected to be closer to sector boundaries the complexity of that 

riiSisSSr- 

more conservative and stated that any thing inside a 25 - 30 mile rJ .n<w» J. ,, y ’ , controllers were a bit 
of the conflict. y ^ ^ 30 mile range to a boundary greatly increases the complexity 

fe daia pSSS W0UM re,Uire m0,e ““ ta H »^“- «* d«s no. show up in 


INT 

8.2 

1.93 

DNS 

8 

1.41 

CAP 

7.3 

2.21 

NBR 

7.2 

2.97 

CRD 

7.2 

1.81 

CPA 

6.7 

1.95 

CoD 

6.3 

2.71 

STR 

6.2 

1.48 

FAC 

6 

2.83 

ANG 

5.8 

2.86 

SEP 

5.7 

2.21 

VAS 

5.2 

2.86 

SUA 

5 

2.26 

VDF 

4.4 

2.50 

PRF 

4.4 

2.07 

WND 

3.8 

1.87 
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t-Test Two-Sample Assuming Equal Variances 
FRX vs. PRX x DNS 



Variable 1 

Variable 2 

Mean 

6 

8 

2 

Variance 

3.77777778 

Observations 

10 

10 

Pooled Variance 

2.88888889 

Hypothesized 

0 


Mean Difference 

dF 

18 


t Stat 

-2.6311741 


Pd'<=t) one-tail 

0.00847468 


t Critical one- tail 

1.73406306 


P( l <=t) two- tail 

0.01694936 


t Critical two-tail 

2.10092367 


t-Test Two-Sample Assuming Equal Variances 

FRX vs. PRX x WND 



Variable 1 

Variable 2 

Mean 

6 

3 8 

Variance 

3.77777778 

3.51111111 

Observations 

10 

10 

Pooled Variance 

3.64444444 

Hypothesized 

0 


Mean Difference 

dF 

18 


t Stat 

2.57686707 


P(T<=t) one- tail 

0.0095001 


t Critical one-tail 

1.73406306 


P(T<=t) two- tail 

0.01900021 


t Critical two-tail 

2.10092367 



t-Test Two-Sample Assuming Equal Variances 
PRX vs. PRX x INT 



Variable 1 

Variable 2 

Mean 

6 

8 2 

Variance 

3.77777778 

3.73333333 

Observations 

10 

10 

Pooled Variance 

3.75555556 

Hypothesized 
Mean Difference 

0 


df 

18 


t Stat 

-2.5384615 


P(T<=t) one-tail 

0.01029512 


t Critical one-tail 

1.73406306 


P(T<=t) two-tail 

0.02059023 


t Critical two-tail 

2.10092367 



sec * or Varies [PRXI .tSbtoJd wi* ^ rf 1 E Wn 7 T‘hM - pmiimU >’ “ 
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Appendix B 


Angle of Convergence in Conflict Situation [ANG] 

Absolute Rating: 6.0 

A measure that examines the predicted angle of conveigence in a conflict. 


J”; general controllers staled that shallower angles of convergence result in a longer period of potential conflict and 
requrre action to be taken sooner Tins, the complexity is increased. However toare 

about various angles worth menti oning - ^ specmcs 


All controllers agree that as angles gei 
presented with a close- to-90” situadoa 




* — w^gvAujr mutu &ieaier man n tUey 


One controller drew cut the following graphic (presented on the left) to describe the relative levels of complexity 
between different angles of convergence. As the numerical value of each section increases (From I to V) the * 
complexity associated with that angle of convergence increases. Angle lines are approximately 30” in separation 
Another control er drew a different graphic to describe the relative levels of complexity associated with vLing ancle 

Ssa *? 1 "■ SLned hum 



^l' CSp0:t “ thne of prediclion for ihese cnollict situations, mosi controllem feel chat the time in minutes 
should correspond wttli fee relative level of complexity. because as fee complexly of .XSlS 
needed to resolve fea, confta mcreases as well. The range of these dmes shookl be from 8 to 


INT 

DNS 

NBR 

CAP 6.6 

VDF 

CoD 

WND 

PRX 

VAS 

SEP 

PRF 

CPA 

STR 

FAC 

CRD 

SUA 


7.1 

3.03 

6.8 

2.39 

6.8 

3.08 


2.59 

6.2 

2.20 

6 

2.45 

5.9 

3.45 

5.8 

2.86 

5.1 

2.47 

5.1 

2.81 

4.7 

2.26 

4.6 

3.37 

4.4 

2.37 

4.3 

2.67 

4.3 

2.91 

3.9 

2.73 


Again, we see the impact that intent information 
[INT] has on the complexity of the situation. This 
time, when the issue of intent is combined with the 
angle of convergence, the complexity is rated fairly 

high due to the fact that the degree of turn required to 

solve a conflict situation is partly determined on the 
knowledge of where the aircraft are going to be in the 
near future. 
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Coordination [CRD] 

Absolute Rating: 6.7 

An examination of the impact that the presence of aircraft that require some form of coordination (with other sectors 
etc.) tor proper control has on the complexity of an air traffic situation. 

Throughout the entire interview process, controllers consistently stated that situations which require coordination 
have the potential to increase the complexity of control. Coordination between sectors can be problematic at times 
but in general it is considered manageable. es ’ 

If one examines the absolute ratings of the combined factors, it can be seen that issues such as density [DNS1 the 
proximity of a conflict to a sector boundary [PRX], and the number of facilities [FAC], when combined with the 
requirement for coordination, result in higher complexity. 


DNS 

7.3 

2.79 

PRX 

7.2 

1.81 

FAC 

7 

3.13 

SEP 

6.4 

2.99 

INT 

6.2 

3.33 

CAP 

6.1 

3.07 

STR 

6 

2.79 

CoD 

6 

3.16 

SUA 

5.6 

3.10 

CPA 

5.6 

2.76 

NBR 

5.4 

2.80 

VDF 

5.2 

2.94 

PRF 

4.5 

3.10 

WND 

4.4 

3.63 

ANG 

4.3 

2.91 

VAS 

4.2 

3.19 
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Appendix B 


Neighbors [NBR] 

Absolute Rating: 6.7 


This factor concerns the distance between 
time. 


aircraft pairs in conflict and other aircraft within some parameter distance or 


Many controllers stated that if neighborine aircraft are within X.in miu ~r o- . , 

associated with the presence of that neighbor ic i, • u u es of a conflict, then the complexity 

stated that the presence of other aircrafftithin 15^0 rSes ^ m0re conservativ e; some 

complexity. ° m,CS of a confljct sltuatlon * enough to impact the 


INT 

DNS 

PRX 

ANG 

CAP 

CoD 

CPA 

SEP 

SUA 

WND 

CRD 

VDF 

STR 

PRF 

VAS 

FAC 


8 

3.09 

7.4 

2.80 

7.2 

2.97 

6.8 

3.08 

6.7 

2.87 

6.6 

3.06 

6.1 

2.56 

5.7 

2.98 

5.5 

2.80 

5.5 

2.76 

5.4 

2.80 

5.3 

2.67 

5.2 

3.05 

4.9 

2.73 

4.8 

3.01 

4.3 

3.09 
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Separation Requirements [SEP] 

Absolute Rating: 6.3 


A measure that examines the impact that imposed 
have on complexity. 


separation requirements for longitudinal sequencing and 


spacing 


r' 0 " * 1 ’ — -*-*■ — w- 


INT 

DNS 

CAP 

CRD 

PRX 

NBR 

CoD 

VAS 

ANG 

CPA 

SUA 

FAC 

PRF 

VDF 

STR 

WND 


7.2 3.05 

6.9 2.96 

6.7 1.77 

6.4 2.99 

5.7 2.21 

5.7 2.98 

5.6 2.07 

5.2 2.25 

5.1 2.81 

5 2.36 

4.9 2.64 

4.9 2.42 

4.9 2.51 

4.8 2.70 

4.4 2.84 

4.3 2.71 
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Variance in Directions of Flight [VDF] 

Absolute Rating: 5.1 

A measurement that looks at the variability of direction for each aircraft to be controlled. 


The controllers agreed that if all of the aircraft within a sector are moving in the “same” direction then the 
complexity associated with the direction of flight is not considered to be an impact 

^e of the possible problems with everyone traveling in the same direction is the shallower angles of converses 


INT 

8 

1.83 

DNS 

6.7 

2.87 

ANG 

6.2 

2.20 

CPA 

5.8 

2.30 

CoD 

5.7 

3.59 

CAP 

5.7 

3.43 

STR 

5.3 

2.58 

WND 

5.3 

2.83 

NBR 

5.3 

2.67 

CRD 

5.2 

2.94 

SEP 

4.8 

2.70 

VAS 

4.6 

2.41 

FAC 

4.5 

2.99 

PRX 

4.4 

2.50 

SUA 

4.3 

2.26 

PRF 

4.3 

1.83 


t-Tesf Two-Sample Assuming Equal Variances 
VDF vs. VDF x INT 



Variable 1 

Variable 2 

Mean 

5.1 

8 

Variance 

4.54444444 

3.33333333 

Observations 

10 

10 

Pooled Variance 

3.93888889 

Hypothesized 
Mean Difference 

0 


df 

18 


t Stat 

-3.2673536 


P(T<=t) one-tail 

0.0021393 


t Critical one-tail 

1.73406306 


P(T<=t) two-tail 

0.00427861 


t Critical two-tail 

2.10092367 
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Airspace Structure [STR] 

Absolute Rating: 5.2 

This measurement will examine the impact that sector size and structure has on the complexity of air traffic control. 


Overall, controllers feel that there is not a big difference between sector shapes with respect to complexity One 
controller stated that larger, more round sectors might be more difficult because of the increased amount of airspace 
.tSr" 0 7 However, for long, narrow sectors, which are smaller in size, such as those typically 

^sociated with arrivals mto an airport, you are required to look farther outside the sector (in terms of time) to pLict 

Kr rflh. ? y ° U „ ^ t0 1110116856 y0Ur 56811 r8te beC8USe you have ,ess time to deal with situations 

Both of these factors can potentially increase the complexity of a given situation. According to the interview 

remits, smaller sectors also require an increase in the amount of communication /coordination needed for effective 
^ ^ ^ h8nd ’ 131861 SeCt ° rS aUow one controUer more time to formulate a plan and see 


For most of the controllers, the difficulty associated with any sector mainly comes from aircraft that are present in 
the sector that are behaving differently from the intended design of the sector. For example, in a long nL>w 
arrival sector ^plexityincreases when traffic crosses “against the grain” of the sector (either directional as in 
north or southbound overflight traffic flowing through an east-west oriented arrival sector, or on an altitude basis 

VOl,me 0f ^ ^ wit tese • 


The complexity associat^ with the airspace structure is probably best evaluated by looking at both the shape of the 

sector hSfm ^ f ^‘ C 8SS0Ciated W ‘ th ±at ****• A number controllers suggested that the design of the 
33 341 m fa f ^ 8 major a * t " butor to complexity. If a sector is set up wrong (for example, sectors 16 

”;i 4) * hen f** co “ plex “y “creases because of the difficulties associated with trying to “correct” for the bad desisn 
of the sector. Another example would be if a really large sector was designed for arrival/departure traffic [CoD] § 


DNS 

CoD 

PRX 

CRD 

INT 

CAP 

CPA 

FAC 

VDF 

NBR 

SUA 

VAS 

ANG 

SEP 

PRF 

WND 


7.1 2.28 

7 2.11 

6.2 1.48 

6 2.79 

5.9 3.35 

5.6 2.37 

5.6 2.32 

5.3 2.75 

5.3 2.58 

5.2 3.05 

5.1 3.03 

4.7 1.64 

4.4 2.37 

4.4 2.84 

3.4 1.78 

3.4 2.91 
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Performance Mix of Traffic [PRF] 

Absolute Rating: 5.1 


A measurement tha, looks a. Ore vurianc in perform* capabilities of cmreu, and expecred utarf,. 

^ocia^ ch "«»n Mi “ « "* aircraft are similar, the complexity 

As a reference point, one controller gave the following “classes" cf jef performance characieristics: 


_ B737 MD80 A3m 

_• B727 A3 10 B757 

• B747 DC10 B777 


B767 

L1011 


(new) MD8Q 


When asked to determine a reasonable amwmt of time to examine the nm-fi™™,,., • rr- 

some controllers felt that anywhere between 15 and 20 minutes would^fhZfiTw * lraffic . ,hat . lS expected - 
primarily on the fact that controllers receive flight cfri™ „ , b ^ hel P ful - ^ information is based 

sector. Two other controllers suggested that looking oaWS-lOr^/ < ? m “ ules besfore 80 aircraft enters their 
based on their 8 minute trend vector line currently available on the FDADs enOUgh t ™ e ' 71x15 number wa « 


INT 

CAP 

CoD 

DNS 

VAS 

CPA 

NBR 

SEP 

WND 

ANG 

CRD 

PRX 

VDF 

FAC 

STR 

SUA 


7.3 

1.89 

7 

1.94 

6.2 

2.62 

5.9 

2.42 

5 

2.36 

4.9 

2.33 

4.9 

2.73 

4.9 

2.51 

4.7 

3.30 

4.7 

2.26 

4.5 

3.10 

4.4 

2.07 

4.3 

1.83 

4.2 

2.70 

3.4 

1.78 

3 

2.00 


t-Test Two-Sample Assuming Equal Variances 
PRF vs. PRF x INT 



Variable 1 

Variable 2 

Mean 

Variance 

5.1 

1 3 

6.32222222 

3.56666667 

Observations 

10 

10 

Pooled Variance 

4.94444444 

Hypothesized 
Mean Difference 

0 


df 

18 


t Stat 

-2.212325 


P(T<=t) one-tail 

0.02005512 


t Critical one-tail 

1.73406306 


P(T<=t) two-tail 

0.04011024 


t Critical two- tail 

2.10092367 
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Special Use Airspace [SUA] 
Absolute Rating: 3.9 


This measure is intended to identify how the number/size/activity of restricted 
airspace impact the complexity of an air traffic scenario. 


areas, warning areas, and military 


Some controllers feel that an SUA located near sector boundaries increases the comnlexiiv a*. ,u^ • 

to ^ boan ^ 


Somewhat contrary to that point, the location of the SUA with respect to the sectnr , 


SStSirS-T* ,r the 

— - Howeve, , 

?^^SSTSSSX- , £ , I 

ftom 15-30 minutes prior to that SUA becoming ach™"^ tSr’ S mT "'T 18 

mrnutes ahead of time. 8 arger 5UAs * “^y w,u start planmng about 45 


DNS 

INT 

CRD 

NBR 

CPA 

STR 

PRX 

CoD 

SEP 

CAP 

FAC 

VDF 

ANG 

VAS 

PRF 

WND 


7.1 

1.60 

6.2 

3.68 

5.6 

3.10 

5.5 

2.80 

5.4 

2.55 

5.1 

3.03 

5 

2.26 

4.9 

3.07 

4.9 

2.64 

4.8 

2.66 

4.3 

3.30 

4.3 

2.26 

3.9 

2.73 

3.6 

2.12 

3 

2.00 

2.3 

2.21 


t-Test: Two-Sample Assuming Equal Variances 
SUA vs. SUA x DNS 



Variable 1 

Variable 2 

Mean 

3.9 

7 i 

Variance 

4.1 

2.54444444 

Observations 

10 

10 

Pooled Variance 

3.32222222 

Hypothesized 
Mean Difference 

0 


df 

18 


t Stat 

-3.9257319 


P(T<=t) one-tail 

0.00049547 


t Critical one-tail 

1.73406306 


P(T<=t) two-tail 

0.00099094 


t Critical two- tail 

2.10092367 
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Number of Facilities [FAC] 

Absolute Rating: 5.0 


A simple count of the number of facilities being seived by. or contained within, the sector. 

AI^CC^ in^^^th^^n^xkyinCTe^es 1 b f. < ^ tr ° UerS “ either or other 

required [CRD], The relationship between the number f f t.* l ° ^ Ulaeased ^ount of coordination that is 
however. In far,, nrany eemrolte sued to, to rtoUonsS k 

open to level of activity to>»ew,toili ty may have depends 

situation becomes. *** more “*"* 0,066 fac *ta are. the more complex the 


Finally, controllers stated that the impact that an increased number of facilities fin t«»rmc «f • . , . 

more apparent in lower altitude sectors For examnle if thZT^iZ TT^ (m terms of ^rts) has is generally 

greatly depends upon whether or ^TcONul f, Z T* ^ ^ 
as u/h#»n on adtpp — , — n_ . - ' a If it is, then the complexity isn’t as bad 


as when an ARTCC stroller also needs to 'control 7pp^^ 


airport. 


CRD 

PRX 

CoD 

INT 

DNS 

STR 

CAP 

CPA 

SEP 

VAS 

VDF 

SUA 

ANG 

NBR 

PRF 

WND 


7 

3.13 

6 

2.83 

5.8 

3.22 

5.6 

3.57 

5.4 

3.57 

5.3 

2.75 

5.3 

3.16 

5.3 

3.13 

4.9 

2.42 

4.8 

2.57 

4.5 

2.99 

4.3 

3.30 

4.3 

2.67 

4.3 

3.09 

4.2 

2.70 

3.7 

3.27 
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Variance in Aircraft Speed [VAS] 

Absolute Rating: 4.3 

A measurement that looks at the variability of speed tracked for each aircraft. 








In general, the controllers agreed that if there was a high level of variance in aircraft speeds, 
that situation would increase, especially with respect to overtakes. 

When asked about the “equality” of speeds, many controllers stated that aircraft speed differences below 20 kts ffor 
are generally considered to be equal, and therefore the complexity associated with 20 kt differences in aircraft 
speeds is fairly low. Some controllers continued, saying that speed differences of 30 - 50 kts generates some 

“ 0USl ' 10 C0 “ Cem - ‘ hM! Speed diSemnals ab °* 50 


An interesting point is the fact that the complexity associated with speed differences depends greatly upon their 
current separation. For example, if two aircraft are 20 miles apart, and the second aircraft has a 20 kt overtake it 
will be quite some time before action needs to be taken. If. however, they are only separated by 8 miles then ’ 
acuoos to prevent an overtake conflict need to be taken more quickly. tea oy 5 miles, then 

Because controllers view aircraft speeds in terms of miles-per-minute. many controllers feel that this ram** „> main c 
^ ^der high and low speeds. However, others feel Z therange difLS“^h[Z s^T 
lal^m h AU . coatro l le ^ a f eed at togher speeds (especially for a head-on conflict situation) aSon !Zt be 
wi,r^r er - "? 11 makes 861156 to look further ahead in time to determine the impact that this factor 

m m ,u *** A few controllers ^ mentioned that at lower altitudes, speed adjustments are easier 

mights aSuyK ^ ClXa " “ or ’ 0mal " I"*)- “1 ta complexity 


INT 

CAP 

DNS 

CoD 

CPA 

PRX 

SEP 

ANG 

PRF 

FAC 

NBR 

STR 

VDF 

WND 

CRD 

SUA 


6.9 

1.97 

6.4 

2.91 

6.3 

2.83 

6 

3.50 

5.4 

2.17 

5.2 

2.86 

5.2 

2.25 

5.1 

2.47 

5 

2.36 

4.8 

2.57 

4.8 

3.01 

4.7 

1.64 

4.6 

2.41 

4.4 

2.72 

4.2 

3.19 

3.6 

2.12 


t-Test Two-Sample Assuming Equal Variances 
VAS vs. VAS x INT 



Variable 1 

Variable 2 

Mean 

4.3 

6.9 

Variance 

5.34444444 

3.87777778 

Observations 

10 

10 

Pooled Variance 

4.61111111 

Hypothesized 
Mean Difference 

0 


df 

18 


t Stat 

-2.7074195 


P(T<=t) one-tail 

0.00721126 


t Critical one-tail 

1.73406306 


P(T<=t) two-tail 

0.01442253 


t Critical two-tail 

2.10092367 
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Winds [WND] 

Absolute Rating: 3.2 


A measure »f the wind speed and azimuth hy altitude. aud to imp*, « ahcnrf, perfommce characteristics. 

3 ■ °r ■ rr 1 10 1 *■ *-*- <* **- - „ „ Ws 
o. atc. jLtS sssasSrS: is** r* *• *r* ^ a — « *«• 



/t 

a ( n.l> 

Somewhat 

>48.3 

14.38 

Significant 

>84.3 

16.18 


^asestsss^ 

on aircraft speeds. One controller stated that crosswindfSe usuaUv ^ 10 ** mpaCt *** Can have 

can actually help in situations that reauirc * h T- “S* y ** f VeCt ° r 0peratl0QS - Fi °ally. headwinds 

than expected, which can create problems. W ^ ercases> ^ headwinds cause aircraft to turn more 


INT 

ANG 

NBR 

VDF 

CAP 

PRF 

VAS 

CPA 

CRD 

CoD 

SEP 

DNS 

PRX 

FAC 

STR 

SUA 


6 

3.83 

5.9 

3.45 

5.5 

2.76 

5.3 

2.83 

5.2 

2.90 

4.7 

3.30 

4.4 

2.72 

4.4 

2.67 

4.4 

3.63 

4.3 

2.21 

4.3 

2.71 

4.1 

3.14 

3.8 

1.87 

3.7 

3.27 

3.4 

2.91 

2.3 

2.21 
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t-Test: Two-Sample Assuming Equal Variances 
WND vs. WND x ANG 



Variable 1 

Variable 2 

Mean 

3.2 

5.9 

Variance 

3.06666667 

11.8777778 

Observations 

10 

10 

Pooled Variance 

7.47222222 

Hypothesized 

0 


Mean Difference 

dr 

18 


t Stat 

-2.2086346 


E(T<=t) one-tail 

0.0202034 


t Critical one-tail 

1.73406306 


P(T<=t) two- tail 

0.04040681 


t Critical two-tail 

2.10092367 


t-Test Two-Sample Assuming Equal Variances 

WND vs. WND x NBR 



Variable 1 

Variable 2 

Mean 

3.2 

5.5 

Variance 

3.06666667 

7.61111111 

Observations 

10 

10 

Pooled Variance 

5.33888889 

Hypothesized 

0 


Mean Difference 

df 

18 


t Stat 

-2.2258065 


P(T<=t) one-tail 

0.01952189 


t Critical one-tail 

1.73406306 


P(T<=t) two-tail 

0.03904379 


t Critical two-tail 

2.10092367 



t-Test Two-Sample Assuming Equal Variances 
WND vs. WND x INT 



Variable 1 

Variable 2 

Mean 

3.2 

6 

Variance 

3.06666667 

14.6666667 

Observations 

10 

10 

Pooled Variance 

8.86666667 

Hypothesized 

0 


Mean Difference 
df 

18 


t Stat 

-2.1026299 


P(T<=t) one-tail 

0.02491647 


t Critical one-tail 

1.73406306 


P(T<=t) two-tail 

0.04983293 


t Critical two- tail 

2.10092367 
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APPENDIX C - MEASUREMENT COMPARISON PLOTS 


INITAL COMPLEXITY ALGORITHM VS. CONTROLLER RATINGS.. 
SECOND COMPLEXITY ALGORITHM VS. CONTROLLER RATINGS 
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